task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-2nc42-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-2nc42-kubeconfig [get-kubeconfig] Found admin password secret: cluster-2nc42-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-2nc42-password [get-kubeconfig] API Server URL: https://ad683cc5765bd41568caf2dbd38220a0-0b39a665eecef20c.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.1843c268-292f-4bb4-8301-17216fecb996.prod.konfluxeaas.com [clone-repo] sync-260716_upstream-to-odh [clone-repo] https://github.com/maskarb/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 91% (2948/3211) Updating files: 92% (2955/3211) Updating files: 93% (2987/3211) Updating files: 94% (3019/3211) Updating files: 95% (3051/3211) Updating files: 96% (3083/3211) Updating files: 97% (3115/3211) Updating files: 98% (3147/3211) Updating files: 99% (3179/3211) Updating files: 100% (3211/3211) Updating files: 100% (3211/3211), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:ab642da407697deef24380a61e5f57cc5e777e6050a63425bb3ba41797861c92 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:ab642da407697deef24380a61e5f57cc5e777e6050a63425bb3ba41797861c92 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:2cd325463239f124d8c2e49d66d19cec3cd81f83d9a406a625c6bf289021745f [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:2cd325463239f124d8c2e49d66d19cec3cd81f83d9a406a625c6bf289021745f [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:1847aecab52b87fbf9dd769d31689e5e091159a8db2cd3d44028fb80f0306eb0 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:1847aecab52b87fbf9dd769d31689e5e091159a8db2cd3d44028fb80f0306eb0 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:cfa83fb6f90f74e7045b0a434888d21d8e8ba8f5198e116818cf8cc9fdf0843e [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:cfa83fb6f90f74e7045b0a434888d21d8e8ba8f5198e116818cf8cc9fdf0843e [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.24) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1749 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1749 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1749 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 274 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.82s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 684ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.14.1 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.4.2 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==49.0.0 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.20.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.11.9 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.13.0 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.7.0 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 46ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://ad683cc5765bd41568caf2dbd38220a0-0b39a665eecef20c.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-77f648d694-d4whs 1/1 Running 0 41s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-77f648d694-d4whs condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-55557df8c-4htwm 1/1 Running 0 46s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-55557df8c-4htwm condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-56f9798bb4-5c95p 1/1 Running 0 51s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-56f9798bb4-5c95p condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:2cd325463239f124d8c2e49d66d19cec3cd81f83d9a406a625c6bf289021745f [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:cfa83fb6f90f74e7045b0a434888d21d8e8ba8f5198e116818cf8cc9fdf0843e [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:ab642da407697deef24380a61e5f57cc5e777e6050a63425bb3ba41797861c92 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:1847aecab52b87fbf9dd769d31689e5e091159a8db2cd3d44028fb80f0306eb0 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:2cd325463239f124d8c2e49d66d19cec3cd81f83d9a406a625c6bf289021745f [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:cfa83fb6f90f74e7045b0a434888d21d8e8ba8f5198e116818cf8cc9fdf0843e [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:ab642da407697deef24380a61e5f57cc5e777e6050a63425bb3ba41797861c92 [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:1847aecab52b87fbf9dd769d31689e5e091159a8db2cd3d44028fb80f0306eb0 [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-uds-tokenizer=quay.io/opendatahub/llm-d-kv-cache:v0.8.0 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-master [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-master# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-565cf7448c-6d7tb 0/1 ContainerCreating 0 7s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-565cf7448c-6d7tb condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.24) [e2e-llm-inference-service] 🎯 Server version (4.21.24) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-dl9mp 1/1 Running 0 13s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-dl9mp condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-94bb8fbfd-qnh6j 0/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-94bb8fbfd-qnh6j condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-769cdd8d88-mfr9j 1/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-c4d5z 1/1 Running 0 67s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-c4d5z condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-c4d5z 1/1 Running 0 77s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-c4d5z condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-5c5755fdcb-8g9zs" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-5c5755fdcb-g2x5r condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.24 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.24 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.24 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.1 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.0 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.0 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.1 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:2cd325463239f124d8c2e49d66d19cec3cd81f83d9a406a625c6bf289021745f [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:2cd325463239f124d8c2e49d66d19cec3cd81f83d9a406a625c6bf289021745f [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:fast [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:4d2ce35df65bab6d2693c9c94b3f24844e28d1e34eb5c52b243f168e1bce3e01 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:cfa83fb6f90f74e7045b0a434888d21d8e8ba8f5198e116818cf8cc9fdf0843e [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:1f2822f55f8b2129ba0b56c19c5a5c4caa2c93f9715c71e2aa381d4b80e67b07 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.122.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'asyncio': '0.23.8', 'anyio': '4.9.0', 'cov': '5.0.0', 'xdist': '3.6.1', 'httpx': '0.30.0', 'metadata': '3.1.1', 'json-report': '1.5.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: asyncio-0.23.8, anyio-4.9.0, cov-5.0.0, xdist-3.6.1, httpx-0.30.0, metadata-3.1.1, json-report-1.5.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [56 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] 2026-07-16 17:28:36.381 6160 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-16 17:28:36.382 6157 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-16 18:00:20.948 6160 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-16 18:00:20.948 6160 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-16 18:00:20.977 6160 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-16 18:00:20.977 6160 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-16 18:00:20.984 6160 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-16 18:00:20.984 6160 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-16 18:00:34.933 6160 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-16 18:00:34.933 6160 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-16 18:00:34.977 6160 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-16 18:00:34.977 6160 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [e2e-llm-inference-service] ==================================== ERRORS ==================================== [e2e-llm-inference-service] __________________ ERROR collecting llmisvc/test_llm_auth.py ___________________ [e2e-llm-inference-service] ImportError while importing test module '/workspace/source/test/e2e/llmisvc/test_llm_auth.py'. [e2e-llm-inference-service] Hint: make sure your test modules/packages have valid Python names. [e2e-llm-inference-service] Traceback: [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule [e2e-llm-inference-service] mod = import_path(self.path, mode=importmode, root=self.config.rootpath) [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/_pytest/pathlib.py:567: in import_path [e2e-llm-inference-service] importlib.import_module(module_name) [e2e-llm-inference-service] /usr/lib64/python3.11/importlib/__init__.py:126: in import_module [e2e-llm-inference-service] return _bootstrap._gcd_import(name[level:], package, level) [e2e-llm-inference-service] :1204: in _gcd_import [e2e-llm-inference-service] ??? [e2e-llm-inference-service] :1176: in _find_and_load [e2e-llm-inference-service] ??? [e2e-llm-inference-service] :1147: in _find_and_load_unlocked [e2e-llm-inference-service] ??? [e2e-llm-inference-service] :690: in _load_unlocked [e2e-llm-inference-service] ??? [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/_pytest/assertion/rewrite.py:186: in exec_module [e2e-llm-inference-service] exec(co, module.__dict__) [e2e-llm-inference-service] llmisvc/test_llm_auth.py:29: in [e2e-llm-inference-service] from .test_llm_inference_service import ( [e2e-llm-inference-service] E ImportError: cannot import name '_collect_diagnostics' from 'e2e.llmisvc.test_llm_inference_service' (/workspace/source/test/e2e/llmisvc/test_llm_inference_service.py) [e2e-llm-inference-service] ___________________ ERROR collecting llmisvc/test_llm_tls.py ___________________ [e2e-llm-inference-service] ImportError while importing test module '/workspace/source/test/e2e/llmisvc/test_llm_tls.py'. [e2e-llm-inference-service] Hint: make sure your test modules/packages have valid Python names. [e2e-llm-inference-service] Traceback: [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule [e2e-llm-inference-service] mod = import_path(self.path, mode=importmode, root=self.config.rootpath) [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/_pytest/pathlib.py:567: in import_path [e2e-llm-inference-service] importlib.import_module(module_name) [e2e-llm-inference-service] /usr/lib64/python3.11/importlib/__init__.py:126: in import_module [e2e-llm-inference-service] return _bootstrap._gcd_import(name[level:], package, level) [e2e-llm-inference-service] :1204: in _gcd_import [e2e-llm-inference-service] ??? [e2e-llm-inference-service] :1176: in _find_and_load [e2e-llm-inference-service] ??? [e2e-llm-inference-service] :1147: in _find_and_load_unlocked [e2e-llm-inference-service] ??? [e2e-llm-inference-service] :690: in _load_unlocked [e2e-llm-inference-service] ??? [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/_pytest/assertion/rewrite.py:186: in exec_module [e2e-llm-inference-service] exec(co, module.__dict__) [e2e-llm-inference-service] llmisvc/test_llm_tls.py:29: in [e2e-llm-inference-service] from .test_llm_inference_service import ( [e2e-llm-inference-service] E ImportError: cannot import name '_collect_diagnostics' from 'e2e.llmisvc.test_llm_inference_service' (/workspace/source/test/e2e/llmisvc/test_llm_inference_service.py) [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] __________ test_flow_control_smoke[flow-control-utilization-detector] __________ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-flow-control-round-robin', 'workload-llmd-simulator'], prompt='KServe... {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] flow_control_auth = None [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-flow-control-round-robin", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="fc-smoke-test", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.flow_control, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="flow-control-utilization-detector", [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-flow-control-concurrency-detector", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="fc-concurrency-test", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.flow_control, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="flow-control-concurrency-detector", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=True, [e2e-llm-inference-service] ) [e2e-llm-inference-service] def test_flow_control_smoke(test_case: TestCase, flow_control_auth): [e2e-llm-inference-service] """Verify that the EPP boots and serves traffic with flow control enabled. [e2e-llm-inference-service] [e2e-llm-inference-service] Sends requests with different fairness IDs, InferenceObjective headers, and [e2e-llm-inference-service] default headers. When a downstream auth provider is available (via the [e2e-llm-inference-service] flow_control_auth fixture), also verifies the auth -> flow control pipeline. [e2e-llm-inference-service] """ [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] [e2e-llm-inference-service] if flow_control_auth: [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] test_case.llm_service.metadata.annotations.update( [e2e-llm-inference-service] flow_control_auth.get("annotations", {}) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response") [e2e-llm-inference-service] > wait_for_model_response(kserve_client, test_case, test_case.wait_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py:126: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'scheduler-fl... {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-07-16T17:29:36.851056', start_time = 1784222976.8514276 [e2e-llm-inference-service] duration = 902.8749163150787, timestamp_end = '2026-07-16T17:44:39.726362' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-flow-control-round-robin', 'workload-llmd-simulator'], prompt='KServe... {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900, extra_headers = None [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1119: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .get_successful_response at 0x7f02eae34c20> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Service returned 401: [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO kserve:conftest.py:40 Logger configured [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig router-managed-fc-smoke-test-39304599 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig router-managed-fc-smoke-test-39304599 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig router-managed-fc-smoke-test-39304599 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig scheduler-flow-control-round-ro-87e79212 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig scheduler-flow-control-round-ro-87e79212 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig scheduler-flow-control-round-ro-87e79212 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig workload-llmd-simulator-fc-smok-80287f19 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-fc-smok-80287f19 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-fc-smok-80287f19 [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:91 Created ConfigMap kserve-ci-e2e-test/gateway-proxy-config [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-16T17:28:36.741632] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-16T17:28:37.063991] end - ✅ in 0.322s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-16T17:28:37.064129] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-16T17:29:04Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/fc-smoke-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T17:29:04Z', 'message': 'Inference Pool kserve-ci-e2e-test/fc-smoke-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T17:29:04Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:29:04Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T17:29:04Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/fc-smoke-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T17:29:04Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/fc-smoke-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T17:29:04Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:29:04Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-16T17:29:26Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T17:29:26Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T17:29:26Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:29:04Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T17:29:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T17:29:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T17:29:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:29:26Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-07-16T17:29:36.850890] end - ✅ in 59.786s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-07-16T17:29:36.851056] start - args=(, TestCase(base_refs=['router-managed', 'scheduler-flow-control-round-robin', 'workload-llmd-simulator'], prompt='KServe is a', service_name='fc-smoke-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f02ebd49ee0>, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:29:36.851434] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:29:36.866371] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:29:41.897185] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:29:41.916000] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:29:46.944524] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:29:46.959742] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:29:51.993640] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:29:52.004588] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:29:57.028567] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:29:57.044028] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:02.071254] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:02.084270] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:07.123057] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:07.144944] end - ✅ in 0.021s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:12.183181] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:12.194122] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:17.223448] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:17.235959] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:22.261200] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:22.271983] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:27.302792] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:27.316156] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:32.343541] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:32.360215] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:37.401979] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:37.413759] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:42.440594] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:42.456194] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:47.485604] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:47.501006] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:52.532867] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:52.557888] end - ✅ in 0.025s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:30:57.585032] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:30:57.609430] end - ✅ in 0.024s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:02.641926] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:02.654771] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:07.684521] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:07.712832] end - ✅ in 0.028s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:12.747375] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:12.758513] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:17.785698] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:17.797025] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:22.820306] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:22.831284] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:27.859518] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:27.871703] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:32.903969] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:32.917141] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:37.946692] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:37.964891] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:42.992307] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:43.016857] end - ✅ in 0.024s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:48.048986] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:48.060030] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:53.088832] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:53.107380] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:31:58.135132] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:31:58.157917] end - ✅ in 0.022s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:03.184619] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:03.201249] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:08.235467] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:08.251100] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:13.285464] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:13.358466] end - ✅ in 0.072s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:18.389305] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:18.405248] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:23.439092] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:23.457614] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:28.484369] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:28.502349] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:33.527896] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:33.541894] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:38.572021] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:38.588215] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:43.621551] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:43.637293] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:48.662874] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:48.673864] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:53.707249] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:53.718944] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:32:58.746595] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:32:58.757656] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:03.789652] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:03.801205] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:08.825016] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:08.841749] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:13.868979] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:13.885952] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:18.913600] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:18.929035] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:23.958749] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:23.972628] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:29.001796] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:29.012238] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:34.047496] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:34.064646] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:39.096714] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:39.124473] end - ✅ in 0.027s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:44.152749] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:44.165052] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:49.190933] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:49.203597] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:54.235633] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:54.250247] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:33:59.278875] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:33:59.292942] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:04.320755] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:04.334198] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:09.356908] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:09.461871] end - ✅ in 0.104s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:14.489577] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:14.507717] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:19.555447] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:19.567537] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:24.598532] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:24.622974] end - ✅ in 0.024s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:29.652380] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:29.665674] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:34.697504] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:34.711195] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:39.738199] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:39.749274] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:44.786475] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:44.859859] end - ✅ in 0.073s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:49.890511] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:49.901350] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:54.928704] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:54.944459] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:34:59.970301] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:34:59.997313] end - ✅ in 0.026s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:35:05.049627] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:35:05.159552] end - ✅ in 0.109s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:35:10.184155] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:35:10.195112] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:35:15.220743] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:35:15.257747] end - ✅ in 0.037s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:35:20.292444] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:35:20.304291] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:35:25.336251] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:35:25.350691] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:35:30.381157] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:35:30.396059] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:35:35.428796] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:35:35.458584] end - ✅ in 0.029s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:35:40.490173] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:35:40.534767] end - ✅ in 0.044s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:35:45.569297] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:35:45.580274] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:35:50.613647] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:35:50.624695] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:35:55.652954] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:35:55.666244] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:00.692636] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:00.703892] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:05.733285] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:05.748898] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:10.772778] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:10.784201] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:15.810604] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:15.826834] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:20.849231] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:20.862541] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:25.889808] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:25.901303] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:30.926190] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:30.937154] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:35.964491] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:35.979782] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:41.011240] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:41.023925] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:46.046314] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:46.057202] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:51.090509] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:51.101044] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:36:56.124190] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:36:56.137762] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:01.167659] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:01.178831] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:06.202236] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:06.221384] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:11.252025] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:11.264827] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:16.293007] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:16.303544] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:21.338246] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:21.352573] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:26.378254] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:26.389316] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:31.413121] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:31.424288] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:36.446267] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:36.458218] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:41.485689] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:41.496705] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:46.526777] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:46.538509] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:51.564742] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:51.575737] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:37:56.602841] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:37:56.613530] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:01.636385] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:01.650053] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:06.674191] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:06.689626] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:11.713369] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:11.724956] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:16.751829] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:16.763092] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:21.787354] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:21.797861] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:26.822481] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:26.835651] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:31.857129] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:31.867438] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:36.889616] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:36.901079] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:41.925451] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:41.935763] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:46.959646] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:46.972534] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:51.995309] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:52.007295] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:38:57.031975] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:38:57.045885] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:02.074040] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:02.085120] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:07.110587] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:07.126913] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:12.152028] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:12.166660] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:17.190503] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:17.213279] end - ✅ in 0.022s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:22.234800] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:22.257455] end - ✅ in 0.022s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:27.279049] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:27.292050] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:32.318341] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:32.332239] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:37.360596] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:37.372841] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:42.400449] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:42.413325] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:47.436063] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:47.451986] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:52.476998] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:52.488621] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:39:57.515077] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:39:57.525195] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:02.550803] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:02.561858] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:07.589928] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:07.616208] end - ✅ in 0.026s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:12.639375] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:12.651204] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:17.677167] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:17.698014] end - ✅ in 0.020s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:22.722425] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:22.734450] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:27.760803] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:27.771765] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:32.798862] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:32.811349] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:37.837092] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:37.851103] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:42.876495] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:42.888295] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:47.915912] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:47.926319] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:52.954090] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:52.966729] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:40:57.991279] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:40:58.002213] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:03.029952] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:03.040682] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:08.064768] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:08.075571] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:13.101978] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:13.116459] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:18.143177] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:18.154292] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:23.179307] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:23.193145] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:28.215936] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:28.237222] end - ✅ in 0.020s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:33.262555] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:33.274170] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:38.305471] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:38.317986] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:43.340684] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:43.354616] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:48.382430] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:48.400857] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:53.425155] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:53.437212] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:41:58.461863] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:41:58.473332] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:03.503270] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:03.516148] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:08.542511] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:08.564868] end - ✅ in 0.022s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:13.594005] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:13.604822] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:18.628424] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:18.639128] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:23.663561] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:23.673731] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:28.700789] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:28.720167] end - ✅ in 0.019s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:33.744091] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:33.755936] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:38.782362] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:38.793195] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:43.817193] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:43.828280] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:48.855803] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:48.866800] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:53.892628] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:53.902301] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:42:58.927415] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:42:58.938389] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:03.967932] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:03.978687] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:09.002865] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:09.014807] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:14.038781] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:14.049213] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:19.074199] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:19.090262] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:24.110374] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:24.121712] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:29.144618] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:29.157197] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:34.182904] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:34.199940] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:39.221871] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:39.266811] end - ✅ in 0.044s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:44.286601] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:44.297862] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:49.322526] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:49.333380] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:54.358367] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:54.369766] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:43:59.395628] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:43:59.405622] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:44:04.428621] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:44:04.445347] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:44:09.468445] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:44:09.483477] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:44:14.507523] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:44:14.519982] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:44:19.549303] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:44:19.562280] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:44:24.584785] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:44:24.595591] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:44:29.621266] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:44:29.631668] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:44:34.657929] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:44:34.668711] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:44:39.694679] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-smoke-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-smoke-test-39304599'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-round-ro-87e79212'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-smok-80287f19'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:44:39.704483] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-smoke-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-07-16T17:44:39.726362] end - ❌ 902.875s: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_flow_control.py:169 [router-managed-scheduler-flow-control-round-robin-workload-llmd-simulator] Failed: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-llmd-simulator1] ______ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='KServe is a', service_name='llmisvc-router-m... {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response from {service_name}") [e2e-llm-inference-service] > wait_for_model_response( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] test_case, [e2e-llm-inference-service] test_case.wait_timeout, [e2e-llm-inference-service] extra_headers=test_case.extra_headers, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:810: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'workload-llm... {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-07-16T17:35:45.768610', start_time = 1784223345.769116 [e2e-llm-inference-service] duration = 901.6529974937439, timestamp_end = '2026-07-16T17:50:47.422116' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='KServe is a', service_name='llmisvc-router-m... {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] extra_headers = {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'} [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1119: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .get_successful_response at 0x7f137d767c40> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Service returned 401: [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig router-managed-llmisvc-router-m-0cc5ee6c in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-router-m-0cc5ee6c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-router-m-0cc5ee6c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-8461fd55 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-8461fd55 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-8461fd55 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-qw-ab631bc0 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-qw-ab631bc0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-qw-ab631bc0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-ef361761 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-ef361761 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-ef361761 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-5aed586c in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-5aed586c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-5aed586c [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-16T17:35:03.859873] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='KServe is a', service_name='llmisvc-router-managed-test-llm-5b1e8f15', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f137dd92fc0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[TestCase(base_refs=['router-managed', 'workload-llmd-simulator', 'model-qwen2.5-0.5b'], prompt='KServe is a', service_name='llmisvc-model-qwen2-5-0-5b-rout-a50492e9', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f137dd93060>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/Qwen/Qwen2.5-0.5B-Instruct'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-qwen2-5-0-5b-rout-a50492e9', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-qw-ab631bc0'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-ef361761'}, [e2e-llm-inference-service] {'name': 'model-qwen2-5-0-5b-llmisvc-mode-5aed586c'}]}, [e2e-llm-inference-service] 'status': None}, model_name='Qwen/Qwen2.5-0.5B-Instruct')], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-16T17:35:03.872632] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-16T17:35:03.956373] end - ✅ in 0.084s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-16T17:35:03.956574] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-16T17:35:14Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T17:35:14Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T17:35:14Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:35:14Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T17:35:14Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T17:35:14Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T17:35:14Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:35:14Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-16T17:35:37Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T17:35:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T17:35:36Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:35:14Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T17:35:37Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T17:35:37Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T17:35:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:35:36Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-07-16T17:35:45.768456] end - ✅ in 41.811s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-07-16T17:35:45.768610] start - args=(, TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='KServe is a', service_name='llmisvc-router-managed-test-llm-5b1e8f15', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f137dd92fc0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[TestCase(base_refs=['router-managed', 'workload-llmd-simulator', 'model-qwen2.5-0.5b'], prompt='KServe is a', service_name='llmisvc-model-qwen2-5-0-5b-rout-a50492e9', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f137dd93060>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/Qwen/Qwen2.5-0.5B-Instruct'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-qwen2-5-0-5b-rout-a50492e9', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-qw-ab631bc0'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-ef361761'}, [e2e-llm-inference-service] {'name': 'model-qwen2-5-0-5b-llmisvc-mode-5aed586c'}]}, [e2e-llm-inference-service] 'status': None}, model_name='Qwen/Qwen2.5-0.5B-Instruct')], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:35:45.769130] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:35:45.778523] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:35:50.806150] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:35:50.816638] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:35:55.847712] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:35:55.858064] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:00.884214] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:00.895133] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:05.916376] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:05.926685] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:10.954624] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:10.970848] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:15.993196] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:16.004848] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:21.028570] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:21.038577] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:26.059593] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:26.069961] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:31.094632] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:31.104958] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:36.129830] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:36.144100] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:41.171533] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:41.193753] end - ✅ in 0.022s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:46.219981] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:46.230479] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:51.251255] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:51.261939] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:36:56.284569] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:36:56.308670] end - ✅ in 0.024s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:01.333034] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:01.343770] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:06.368307] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:06.383440] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:11.407864] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:11.418587] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:16.445183] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:16.455628] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:21.482384] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:21.495708] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:26.515770] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:26.526498] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:31.558239] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:31.569216] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:36.592641] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:36.608744] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:41.635615] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:41.645915] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:46.670642] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:46.681673] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:51.704546] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:51.715833] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:37:56.741828] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:37:56.757367] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:01.779465] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:01.789304] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:06.813113] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:06.823523] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:11.844843] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:11.855513] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:16.879693] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:16.890629] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:21.918240] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:21.931372] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:26.952635] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:26.964555] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:31.989166] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:31.999084] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:37.026109] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:37.036754] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:42.057868] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:42.069116] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 2: restarts={"openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-16T17:28:58Z\"}}"} resources={"httproutes": 2, "llminferenceservices": 2, "authpolicies": 1} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:47.094935] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:47.107536] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:52.133562] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:52.150096] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:38:57.174337] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:38:57.191780] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:02.216692] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:02.227211] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:07.253331] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:07.268843] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:12.296352] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:12.309230] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:17.330538] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:17.348025] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:22.375151] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:22.391592] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:27.418931] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:27.431033] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:32.456923] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:32.468460] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:37.489843] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:37.500481] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:42.523504] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:42.534389] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:47.560080] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:47.572783] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:52.594736] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:52.605118] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:39:57.628086] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:39:57.638743] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:02.662614] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:02.673902] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:07.696055] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:07.707814] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:12.731310] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:12.748255] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:17.771362] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:17.788258] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:22.809575] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:22.823780] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:27.845702] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:27.859255] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:32.879684] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:32.889913] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:37.914268] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:37.931501] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:42.950065] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:42.960693] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:47.988723] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:48.000830] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:53.024998] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:53.038372] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:40:58.062125] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:40:58.074606] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:03.093106] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:03.105992] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:08.132261] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:08.145728] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:13.171452] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:13.183498] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:18.208512] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:18.222689] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:23.246957] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:23.257704] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:28.282935] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:28.303869] end - ✅ in 0.021s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:33.326512] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:33.337483] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:38.359371] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:38.375995] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:43.396629] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:43.408134] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:48.432065] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:48.447490] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:53.471075] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:53.481662] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:41:58.502782] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:41:58.513847] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:03.535360] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:03.547600] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:08.567754] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:08.580195] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:13.601177] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:13.612132] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:18.634044] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:18.649531] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:23.678011] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:23.690138] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:28.713579] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:28.726025] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:33.749200] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:33.759644] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:38.783728] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:38.797853] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:43.822276] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:43.832875] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:48.860373] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:48.870037] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:53.891477] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:53.901996] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:42:58.927677] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:42:58.938579] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:03.968046] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:03.979568] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:09.004272] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:09.015458] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:14.037480] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:14.048276] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:19.073787] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:19.089113] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:24.114382] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:24.124752] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:29.145918] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:29.159659] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:34.184508] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:34.199045] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:39.230315] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:39.259723] end - ✅ in 0.029s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:44.287940] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:44.297774] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 3: restarts={"openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-16T17:28:58Z\"}}"} resources={"httproutes": 2, "llminferenceservices": 2, "authpolicies": 1} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:49.322254] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:49.333584] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:54.359322] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:54.369263] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:43:59.391618] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:43:59.401484] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:04.422441] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:04.435705] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:09.461010] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:09.472768] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:14.492032] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:14.502905] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:19.525237] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:19.537904] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:24.562341] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:24.573014] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:29.597696] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:29.608286] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:34.633266] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:34.643720] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:39.670258] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:39.680344] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:44.699587] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:44.709688] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:49.732679] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:49.757599] end - ✅ in 0.024s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:54.782335] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:54.857648] end - ✅ in 0.075s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:44:59.885385] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:44:59.897688] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:45:04.922267] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:45:04.933183] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:45:09.958769] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:45:09.969154] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:45:14.997954] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:45:15.007844] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:45:20.033944] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:45:20.044362] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:45:25.070851] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:45:25.080771] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:45:30.105510] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:45:30.116252] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:45:35.141627] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:45:35.161218] end - ✅ in 0.019s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:45:40.188088] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:45:40.202716] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:45:45.226963] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:45:45.236829] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:45:50.265833] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:45:50.276480] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:45:55.301845] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:45:55.312090] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:00.336001] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:00.345873] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:05.374220] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:05.384019] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:10.405222] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:10.414903] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:15.437180] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:15.447010] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:20.475568] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:20.486613] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:25.509819] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:25.523813] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:30.545216] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:30.555862] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:35.577053] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:35.587781] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:40.610368] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:40.620337] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:45.640593] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:45.651013] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:50.676973] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:50.687007] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:46:55.713269] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:46:55.723695] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:00.745312] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:00.757430] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:05.779231] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:05.789000] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:10.813602] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:10.823756] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:15.843013] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:15.853100] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:20.882129] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:20.891806] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:25.914841] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:25.925163] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:30.949104] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:30.958715] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:35.988712] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:35.999825] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:41.025899] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:41.036060] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:46.059198] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:46.069604] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:51.096835] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:51.107241] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:47:56.133791] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:47:56.143664] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:01.165231] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:01.180304] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:06.209991] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:06.220691] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:11.243457] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:11.253022] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:16.279226] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:16.289524] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:21.312830] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:21.325375] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:26.352037] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:26.367344] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:31.391991] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:31.402556] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:36.431276] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:36.444005] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:41.468084] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:41.479669] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:46.504106] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:46.514451] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 4: restarts={"openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-16T17:28:58Z\"}}"} resources={"httproutes": 3, "llminferenceservices": 3, "authpolicies": 1} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:51.540089] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:51.550943] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:48:56.573697] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:48:56.584679] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:01.610932] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:01.624849] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:06.649225] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:06.660108] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:11.685065] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:11.699915] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:16.724504] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:16.735522] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:21.761869] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:21.772133] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:26.801790] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:26.812834] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:31.835992] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:31.846577] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:36.877424] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:36.887813] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:41.909005] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:41.921001] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:46.942159] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:46.953379] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:51.976583] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:51.987149] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:49:57.009291] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:49:57.019304] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:50:02.048292] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:50:02.059465] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:50:07.082719] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:50:07.092593] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:50:12.117730] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:50:12.134935] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:50:17.158368] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:50:17.173182] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:50:22.202488] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:50:22.217993] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:50:27.245070] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:50:27.254957] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:50:32.274729] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:50:32.287380] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:50:37.312067] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:50:37.321834] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:50:42.346509] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:50:42.358027] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:50:47.386330] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-5b1e8f15', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-0cc5ee6c'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-8461fd55'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-5b1e8f15: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:50:47.396423] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-07-16T17:50:47.422116] end - ❌ 901.653s: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:825 [router-managed-workload-llmd-simulator] ❌ ERROR: Failed to call llm inference service llmisvc-router-managed-test-llm-5b1e8f15: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-router-managed-test-llm-5b1e8f15' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:03Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:03Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:03Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:35:45Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '27721' [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-router-m-0cc5ee6c [e2e-llm-inference-service] - name: workload-llmd-simulator-llmisvc-8461fd55 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:37Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:36Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:36Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:14Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:45Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:45Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:45Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:36Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-concurrency-test-kserve-b5d44467c-xdjqc to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-concurrency-test-kserve-b5d44467c-xdjqc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-concurrency-test-kserve-router-scheduler-78c47c8645 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-concurrency-test-kserve-b5d44467c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/fc-concurrency-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-concurrency-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-concurrency-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/fc-concurrency-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-fc-concurrency-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/fc-concurrency-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/fc-concurrency-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-concurrency-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-concurrency-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/fc-concurrency-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:45:18 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-concurrency-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/fc-concurrency-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/fc-concurrency-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:45:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [fc-concurrency-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-smoke-test-kserve-7f4687897-r5tjx to ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" in 1.6s (1.6s including waiting). Image size: 79178094 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-smoke-test-kserve-7f4687897-r5tjx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.25/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:02 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" in 1.538s (1.538s including waiting). Image size: 179512283 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-smoke-test-kserve-router-scheduler-fcbfbd8bf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-smoke-test-kserve-7f4687897 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/fc-smoke-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-smoke-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-smoke-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/fc-smoke-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-fc-smoke-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/fc-smoke-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/fc-smoke-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-smoke-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-smoke-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/fc-smoke-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:36 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with scheduler-v06-pd-migration-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/fc-smoke-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/fc-smoke-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [fc-smoke-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:57 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with scheduler-v06-threshold-migration-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with precise-prefix-cache-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:13 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with llmisvc-router-managed-test-llm-e45d1f79 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:15 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:50 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2 to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:08 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:45 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:45 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:49 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-sskh8 to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:44 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:44 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-sskh8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:05 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:43 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:43 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexb8dr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5657fd78d9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexb8dr to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:44 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-69b67488bb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d6c8b5f7c-kgjrz to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" in 4.327s (4.327s including waiting). Image size: 300595129 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:02 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:02 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:02 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d6c8b5f7c-nmsvz to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6d6c8b5f7c-nmsvz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6d6c8b5f7c-kgjrz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-75779d4bffhzf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75779d4bffhzf to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" in 4.016s (4.016s including waiting). Image size: 300595129 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:04 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:05 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:05 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:59 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/llm-d-kv-cache:v0.8.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:43 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/llm-d-kv-cache:v0.8.0" in 43.867s (43.867s including waiting). Image size: 2935159967 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:43 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:43 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.54:8082/healthz": dial tcp 10.133.0.54:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:04 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning BackOff Back-off restarting failed container main in pod precise-prefix-cache-test-kserve-router-scheduler-75779d4bffhzf_kserve-ci-e2e-test(9ff96acc-75a8-48e2-9709-8abb012acf3f) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:15 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.133.0.54:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: Get "http://10.133.0.54:8082/healthz": dial tcp 10.133.0.54:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:48 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-75779d4b5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-6d6c8b5f7c from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:39 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/precise-prefix-cache-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:38 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-68f9854ff-82kzx to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.24/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" in 2.361s (2.361s including waiting). Image size: 166803393 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:51 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" in 1.483s (1.483s including waiting). Image size: 79178094 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-68f9854ff-82kzx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-4cx7q to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:49 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" in 1.754s (1.754s including waiting). Image size: 79178094 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:49 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:49 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.49:8000/health": dial tcp 10.133.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-4cx7q [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-router-scheduler-79ww96l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-79ww96l to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:48 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" in 2.55s (2.55s including waiting). Image size: 179512283 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:42 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-router-scheduler-79c5847dfd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-68f9854ff from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-pd-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-pd-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:29 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/scheduler-v06-pd-migration-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:28 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-v06-pd-migration-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5d54f65f57-27ctz to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.26/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-5d54f65f57-27ctz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-prefill-b58fmzjxs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-b58fmzjxs to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-prefill-b58fbd55d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6ws58 to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedToRetrieveImagePullSecret Unable to retrieve some image pull secrets (scheduler-v06-threshold-migration-test-epp-sa-dockercfg-klsww); attempting to pull the image may not succeed. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:38 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-router-sched6ws58 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-router-scheduler-5db9bd4595 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-5d54f65f57 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-threshold-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-threshold-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:29 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/scheduler-v06-threshold-migration-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:29 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-v06-threshold-migration-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 I0716 17:35:12.395914 1 config.go:808] "Configuration:" =< [e2e-llm-inference-service] { [e2e-llm-inference-service] "DisplayModelName": "facebook/opt-125m", [e2e-llm-inference-service] "IP": "", [e2e-llm-inference-service] "PodName": "", [e2e-llm-inference-service] "PodNameSpace": "", [e2e-llm-inference-service] "VllmDevMode": false, [e2e-llm-inference-service] "block-size": 16, [e2e-llm-inference-service] "data-parallel-rank": -1, [e2e-llm-inference-service] "data-parallel-size": 1, [e2e-llm-inference-service] "dataset-in-memory": false, [e2e-llm-inference-service] "dataset-path": "", [e2e-llm-inference-service] "dataset-table-name": "llmd", [e2e-llm-inference-service] "dataset-url": "", [e2e-llm-inference-service] "default-embedding-dimensions": 384, [e2e-llm-inference-service] "ec-transfer-config": "", [e2e-llm-inference-service] "enable-kvcache": false, [e2e-llm-inference-service] "enable-prefix-caching": false, [e2e-llm-inference-service] "enable-request-id-headers": false, [e2e-llm-inference-service] "enable-sleep-mode": false, [e2e-llm-inference-service] "enforce-eager": false, [e2e-llm-inference-service] "event-batch-size": 16, [e2e-llm-inference-service] "failure-injection-rate": 0, [e2e-llm-inference-service] "failure-types": null, [e2e-llm-inference-service] "fake-metrics": null, [e2e-llm-inference-service] "fake-metrics-refresh-interval": "100ms", [e2e-llm-inference-service] "force-dummy-tokenizer": true, [e2e-llm-inference-service] "global-cache-hit-threshold": 0, [e2e-llm-inference-service] "hash-seed": "", [e2e-llm-inference-service] "image-emission-rate": 0, [e2e-llm-inference-service] "inter-token-latency": "0s", [e2e-llm-inference-service] "inter-token-latency-std-dev": "0s", [e2e-llm-inference-service] "kv-cache-size": 1024, [e2e-llm-inference-service] "kv-cache-transfer-latency": "0s", [e2e-llm-inference-service] "kv-cache-transfer-latency-std-dev": "0s", [e2e-llm-inference-service] "kv-cache-transfer-time-per-token": "0s", [e2e-llm-inference-service] "kv-cache-transfer-time-std-dev": "0s", [e2e-llm-inference-service] "latency-calculator": "", [e2e-llm-inference-service] "log-http": false, [e2e-llm-inference-service] "lora-modules": null, [e2e-llm-inference-service] "max-cpu-loras": 1, [e2e-llm-inference-service] "max-loras": 1, [e2e-llm-inference-service] "max-model-len": 1024, [e2e-llm-inference-service] "max-num-seqs": 5, [e2e-llm-inference-service] "max-request-body-size-mb": 4, [e2e-llm-inference-service] "max-tool-call-array-param-length": 5, [e2e-llm-inference-service] "max-tool-call-integer-param": 100, [e2e-llm-inference-service] "max-tool-call-number-param": 100, [e2e-llm-inference-service] "max-waiting-queue-length": 1000, [e2e-llm-inference-service] "min-tool-call-array-param-length": 1, [e2e-llm-inference-service] "min-tool-call-integer-param": 0, [e2e-llm-inference-service] "min-tool-call-number-param": 0, [e2e-llm-inference-service] "mm-encoder-only": false, [e2e-llm-inference-service] "mm-processor-kwargs": "", [e2e-llm-inference-service] "mm-render-timeout": "1m0s", [e2e-llm-inference-service] "mode": "random", [e2e-llm-inference-service] "model": "facebook/opt-125m", [e2e-llm-inference-service] "object-tool-call-not-required-field-probability": 50, [e2e-llm-inference-service] "omni": false, [e2e-llm-inference-service] "port": 8000, [e2e-llm-inference-service] "prefill-overhead": "0s", [e2e-llm-inference-service] "prefill-time-per-token": "0s", [e2e-llm-inference-service] "prefill-time-std-dev": "0s", [e2e-llm-inference-service] "render-timeout": "30s", [e2e-llm-inference-service] "render-url": "http://localhost:8082", [e2e-llm-inference-service] "seed": 1784223312395197000, [e2e-llm-inference-service] "self-signed-certs": false, [e2e-llm-inference-service] "served-model-name": [ [e2e-llm-inference-service] "facebook/opt-125m" [e2e-llm-inference-service] ], [e2e-llm-inference-service] "ssl-certfile": "/var/run/kserve/tls/tls.crt", [e2e-llm-inference-service] "ssl-keyfile": "/var/run/kserve/tls/tls.key", [e2e-llm-inference-service] "startup-duration": "0s", [e2e-llm-inference-service] "time-factor-under-load": 1, [e2e-llm-inference-service] "time-to-first-token": "0s", [e2e-llm-inference-service] "time-to-first-token-std-dev": "0s", [e2e-llm-inference-service] "time-to-generate-image": "0s", [e2e-llm-inference-service] "time-to-generate-image-std-dev": "0s", [e2e-llm-inference-service] "tool-call-not-required-param-probability": 50, [e2e-llm-inference-service] "use-vllm-map-event-format": false, [e2e-llm-inference-service] "zmq-endpoint": "tcp://127.0.0.1:5557" [e2e-llm-inference-service] } [e2e-llm-inference-service] > [e2e-llm-inference-service] I0716 17:35:12.395999 1 tokenizer.go:60] "Force dummy tokenizer flag is set, using simulated tokenizer" model="facebook/opt-125m" [e2e-llm-inference-service] I0716 17:35:12.401463 1 context.go:216] "No dataset path or URL provided, using random text for responses" [e2e-llm-inference-service] I0716 17:35:12.401675 1 communication.go:66] "Starting communication layer" [e2e-llm-inference-service] I0716 17:35:12.401842 1 simulator.go:210] "Start processing routine" [e2e-llm-inference-service] I0716 17:35:12.402080 1 http_server_tls.go:59] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key" [e2e-llm-inference-service] I0716 17:35:12.402146 1 grpc.go:129] "Server starting" protocol="gRPC" port=8000 [e2e-llm-inference-service] I0716 17:35:12.402768 1 http.go:117] "Server starting" protocol="HTTPS" port=8000 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1784223312.6456764,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1784223312.645801,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n weight: 2\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-router-managed-test-llm-5b1e8f15-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1784223312.64591,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.646366,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.646408,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: prefix-cache-scorer} {Type: no-hit-lru-scorer} {Type: max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: no-hit-lru-scorer, Weight: 2.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6464198,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6466732,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"no-hit-lru-scorer","parameters":null,"type":"no-hit-lru-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"no-hit-lru-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1784223312.667069,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1784223312.667218,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6672819,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6673646,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6674948,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6675332,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6675825,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: no-hit-lru-scorer, Type: no-hit-lru-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: no-hit-lru-scorer, Weight: 2.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6676426,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1784223312.667698,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6677122,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"no-hit-lru-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6677418,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6678393,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6678524,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc00072c280 DataConfig:{Sources:[{Plugin:0xc0005805a0 Extractors:[0xc00072c8c0]}]} FlowControlConfig: ParserRegistry:0xc00072cf40}"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6855187,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.685549,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6855626,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.685567,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6855714,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6855755,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6855798,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.685584,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6855881,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6855934,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6855974,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6856012,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6856081,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.685613,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6856391,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6856468,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6856616,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6856742,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6856828,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6857543,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1784223312.685771,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6858056,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6859722,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6862135,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6863215,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.68635,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6864352,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6865723,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.686612,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6870942,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.687179,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6914635,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6914651,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.6917198,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.692244,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.7874415,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.7874742,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784223312.7874415,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.787528,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784223312.887835,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.8878703,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784223312.8878405,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1784223312.887894,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784223312.888033,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-router-managed-test-llm-5b1e8f15-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-5b1e8f15-inference-pool","reconcileID":"4e5d2ef4-0e3f-4c7a-9527-17eb427ef86c"} [e2e-llm-inference-service] {"level":"info","ts":1784223322.8416648,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2","reconcileID":"5f8d23c1-4e78-4e01-8afd-f97c94c4546a"} [e2e-llm-inference-service] {"level":"info","ts":1784223335.4377065,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-router-managed-test-llm-5b1e8f15-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-5b1e8f15-inference-pool","reconcileID":"6a790a19-8c9f-43d3-b355-7d8b6de79e45"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: fa44fa00-939f-4788-aca7-146b2451ccc9 [e2e-llm-inference-service] resourceVersion: '27711' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.29 [e2e-llm-inference-service] nodeName: ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk [e2e-llm-inference-service] uid: 25efbac3-e5a8-4520-8538-a2e5b11b616c [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 325f1511-4bee-4f35-aa4b-5ea500a06d25 [e2e-llm-inference-service] resourceVersion: '27466' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.56 [e2e-llm-inference-service] nodeName: ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2 [e2e-llm-inference-service] uid: ae64d07c-dbee-4be2-936d-7efc5dfdbd89 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2 [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ae64d07c-dbee-4be2-936d-7efc5dfdbd89 [e2e-llm-inference-service] resourceVersion: '27465' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: df5b5c977 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.56/23"],"mac_address":"0a:58:0a:85:00:38","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.56/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.56\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:38\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977 [e2e-llm-inference-service] uid: d1aebf4a-268b-4b5f-a9fe-3297c277c64b [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-132-130 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d1aebf4a-268b-4b5f-a9fe-3297c277c64b"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.56"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-v9xst [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-v9xst [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] hostIP: 10.0.132.130 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.132.130 [e2e-llm-inference-service] podIP: 10.133.0.56 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.56 [e2e-llm-inference-service] startTime: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim-dev@sha256:151063768d4a46d17a30ff512d74946b20cf653a2302601a6a823727fbbe4026 [e2e-llm-inference-service] containerID: cri-o://b7c49ccdd97ff433045f7776c4a93751b8c6cdb19523a54b63d731b09f3f21d9 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-v9xst [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk [e2e-llm-inference-service] generateName: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-69b67488bb- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 25efbac3-e5a8-4520-8538-a2e5b11b616c [e2e-llm-inference-service] resourceVersion: '27709' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 69b67488bb [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.29/23"],"mac_address":"0a:58:0a:84:00:1d","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.29/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.29\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:1d\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-69b67488bb [e2e-llm-inference-service] uid: 4d5cc0f6-3d48-4b09-bc8a-d59940bf852d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-42 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4d5cc0f6-3d48-4b09-bc8a-d59940bf852d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.29"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-ftvxf [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-ftvxf [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-qkt7r [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:13Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] hostIP: 10.0.128.42 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.42 [e2e-llm-inference-service] podIP: 10.132.0.29 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.29 [e2e-llm-inference-service] startTime: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://0a45419ccba92450df1e0ec5067d5164f84df1d168d276aab4dd0767a7d4e9f0 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-ftvxf [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4732e183-a8a1-41b4-84e1-320b4c5844a3 [e2e-llm-inference-service] resourceVersion: '27203' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-qkt7r [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-qkt7r"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-sdm8r"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-qkt7r [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-qkt7r [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: bdc56fe1-b6e5-40a9-a5dc-d802c1805e1e [e2e-llm-inference-service] resourceVersion: '27223' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.6.249 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.6.249 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0b67dd87-a1c1-45be-82c7-34895817e90f [e2e-llm-inference-service] resourceVersion: '27187' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.91.224 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.91.224 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a4dca391-3b59-48ba-a07b-e4312bd8649d [e2e-llm-inference-service] resourceVersion: '27469' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cbcc30a7-3fbe-41ef-a739-18d719bcb5cc [e2e-llm-inference-service] resourceVersion: '27713' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n- type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ no-hit-lru-scorer\n weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-69b67488bb" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d1aebf4a-268b-4b5f-a9fe-3297c277c64b [e2e-llm-inference-service] resourceVersion: '27468' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: df5b5c977 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] uid: a4dca391-3b59-48ba-a07b-e4312bd8649d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a4dca391-3b59-48ba-a07b-e4312bd8649d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: df5b5c977 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: df5b5c977 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-69b67488bb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4d5cc0f6-3d48-4b09-bc8a-d59940bf852d [e2e-llm-inference-service] resourceVersion: '27712' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 69b67488bb [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] uid: cbcc30a7-3fbe-41ef-a739-18d719bcb5cc [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"cbcc30a7-3fbe-41ef-a739-18d719bcb5cc"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 69b67488bb [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 69b67488bb [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n- type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ no-hit-lru-scorer\n weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 98b09ad1-f68e-4790-864e-54a1e319a021 [e2e-llm-inference-service] resourceVersion: '27213' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f7a0e4f1-d45e-49da-98d2-d34758490f8f [e2e-llm-inference-service] resourceVersion: '27211' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service-7xk92 [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-5b1e8f15-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8a775406-8649-42cb-99c8-abad416d5259 [e2e-llm-inference-service] resourceVersion: '27710' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] uid: bdc56fe1-b6e5-40a9-a5dc-d802c1805e1e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"bdc56fe1-b6e5-40a9-a5dc-d802c1805e1e"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.29 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk [e2e-llm-inference-service] uid: 25efbac3-e5a8-4520-8538-a2e5b11b616c [e2e-llm-inference-service] nodeName: ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-s66ktq [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8475ca32-2dae-4358-bfcb-ba10a442f37b [e2e-llm-inference-service] resourceVersion: '27467' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] uid: 0b67dd87-a1c1-45be-82c7-34895817e90f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0b67dd87-a1c1-45be-82c7-34895817e90f"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.56 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2 [e2e-llm-inference-service] uid: ae64d07c-dbee-4be2-936d-7efc5dfdbd89 [e2e-llm-inference-service] nodeName: ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 98b09ad1-f68e-4790-864e-54a1e319a021 [e2e-llm-inference-service] resourceVersion: '27213' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f7a0e4f1-d45e-49da-98d2-d34758490f8f [e2e-llm-inference-service] resourceVersion: '27211' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:35:11Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:35:36Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27613' [e2e-llm-inference-service] uid: 68279b13-4462-4687-a5b1-8b38fbcb5a4c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:13Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:35:36Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27613' [e2e-llm-inference-service] uid: 68279b13-4462-4687-a5b1-8b38fbcb5a4c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:13Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27595' [e2e-llm-inference-service] uid: 37fb936b-e002-4517-8cc6-44ac6386c845 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:14Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:35:15Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27379' [e2e-llm-inference-service] uid: 74f99bd0-ac16-4e7c-bbed-a898583ed411 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:14Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:35:15Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27260' [e2e-llm-inference-service] uid: d2124a7a-d634-49fe-9046-d308e57be2a2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27604' [e2e-llm-inference-service] uid: f2a294bc-279d-4e8b-b6f8-2a1bf1a1a121 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-inference--ip-33d4275d.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:13Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:13Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27295' [e2e-llm-inference-service] uid: 320ff62a-4524-4815-b8ed-c137a1e6d139 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27260' [e2e-llm-inference-service] uid: d2124a7a-d634-49fe-9046-d308e57be2a2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27604' [e2e-llm-inference-service] uid: f2a294bc-279d-4e8b-b6f8-2a1bf1a1a121 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-inference--ip-33d4275d.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:13Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:13Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27295' [e2e-llm-inference-service] uid: 320ff62a-4524-4815-b8ed-c137a1e6d139 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27260' [e2e-llm-inference-service] uid: d2124a7a-d634-49fe-9046-d308e57be2a2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:35Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27604' [e2e-llm-inference-service] uid: f2a294bc-279d-4e8b-b6f8-2a1bf1a1a121 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-inference--ip-33d4275d.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:13Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:13Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27295' [e2e-llm-inference-service] uid: 320ff62a-4524-4815-b8ed-c137a1e6d139 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"985eb214-8565-4d37-9de8-3c953647bdf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:35:12Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] uid: 985eb214-8565-4d37-9de8-3c953647bdf3 [e2e-llm-inference-service] resourceVersion: '27235' [e2e-llm-inference-service] uid: 3de24bde-99e2-427d-98e8-730236a81c55 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:50:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: df5b5c977 [e2e-llm-inference-service] timestamp: '2026-07-16T17:50:32Z' [e2e-llm-inference-service] window: 11.059s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 18559182n [e2e-llm-inference-service] memory: 24568Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:50:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-5b1e8f15 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 69b67488bb [e2e-llm-inference-service] timestamp: '2026-07-16T17:50:24Z' [e2e-llm-inference-service] window: 15.273s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 8980095n [e2e-llm-inference-service] memory: 34368Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-16T17:50:49.577153] end - ❌ 945.717s: Service returned 401: [e2e-llm-inference-service] __________ test_flow_control_smoke[flow-control-concurrency-detector] __________ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-flow-control-concurrency-detector', 'workload-llmd-simulator'], promp... {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] flow_control_auth = None [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-flow-control-round-robin", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="fc-smoke-test", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.flow_control, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="flow-control-utilization-detector", [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-flow-control-concurrency-detector", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="fc-concurrency-test", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.flow_control, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="flow-control-concurrency-detector", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=True, [e2e-llm-inference-service] ) [e2e-llm-inference-service] def test_flow_control_smoke(test_case: TestCase, flow_control_auth): [e2e-llm-inference-service] """Verify that the EPP boots and serves traffic with flow control enabled. [e2e-llm-inference-service] [e2e-llm-inference-service] Sends requests with different fairness IDs, InferenceObjective headers, and [e2e-llm-inference-service] default headers. When a downstream auth provider is available (via the [e2e-llm-inference-service] flow_control_auth fixture), also verifies the auth -> flow control pipeline. [e2e-llm-inference-service] """ [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] [e2e-llm-inference-service] if flow_control_auth: [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] test_case.llm_service.metadata.annotations.update( [e2e-llm-inference-service] flow_control_auth.get("annotations", {}) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response") [e2e-llm-inference-service] > wait_for_model_response(kserve_client, test_case, test_case.wait_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py:126: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'scheduler-fl... {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-07-16T17:45:18.683359', start_time = 1784223918.6842113 [e2e-llm-inference-service] duration = 902.1318068504333, timestamp_end = '2026-07-16T18:00:20.816021' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-flow-control-concurrency-detector', 'workload-llmd-simulator'], promp... {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900, extra_headers = None [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1119: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .get_successful_response at 0x7f02eae36200> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Service returned 401: [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig router-managed-fc-concurrency-t-25c29f3a in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig router-managed-fc-concurrency-t-25c29f3a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig router-managed-fc-concurrency-t-25c29f3a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig scheduler-flow-control-concurre-34e50aeb in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig scheduler-flow-control-concurre-34e50aeb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig scheduler-flow-control-concurre-34e50aeb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig workload-llmd-simulator-fc-conc-1c5a0c75 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-fc-conc-1c5a0c75 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-fc-conc-1c5a0c75 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-16T17:44:40.020365] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-16T17:44:40.168364] end - ✅ in 0.148s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-16T17:44:40.168541] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-16T17:44:48Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/fc-concurrency-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T17:44:48Z', 'message': 'Inference Pool kserve-ci-e2e-test/fc-concurrency-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T17:44:48Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:44:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T17:44:48Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/fc-concurrency-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T17:44:48Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/fc-concurrency-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T17:44:48Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:44:48Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-16T17:44:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T17:44:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T17:44:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:44:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T17:44:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T17:44:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T17:44:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:44:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-16T17:44:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T17:44:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T17:44:57Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:44:48Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T17:44:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T17:44:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T17:44:54Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:44:57Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-07-16T17:45:18.683202] end - ✅ in 38.514s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-07-16T17:45:18.683359] start - args=(, TestCase(base_refs=['router-managed', 'scheduler-flow-control-concurrency-detector', 'workload-llmd-simulator'], prompt='KServe is a', service_name='fc-concurrency-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f02ebd49f80>, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:45:18.684222] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:45:18.692597] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:45:23.721119] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:45:23.731057] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:45:28.756918] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:45:28.768101] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:45:33.795079] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:45:33.805569] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:45:38.828932] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:45:38.840964] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:45:43.865481] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:45:43.877268] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:45:48.897482] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:45:48.907061] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:45:53.931116] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:45:53.940910] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:45:58.965340] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:45:58.974839] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:03.998939] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:04.008772] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:09.033609] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:09.043765] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:14.071038] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:14.080485] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:19.105487] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:19.118495] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:24.142295] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:24.153025] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:29.175749] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:29.185279] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:34.206525] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:34.216634] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:39.263329] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:39.276509] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:44.301789] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:44.312821] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:49.335666] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:49.345230] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:54.367313] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:54.376750] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:46:59.401946] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:46:59.411318] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:04.437493] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:04.447789] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:09.475186] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:09.484998] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:14.509895] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:14.519903] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:19.547235] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:19.556995] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:24.584726] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:24.595632] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:29.619947] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:29.629938] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:34.651563] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:34.663190] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:39.683719] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:39.693428] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:44.716320] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:44.725947] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:49.753082] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:49.763125] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:54.787639] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:54.797178] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:47:59.820204] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:47:59.830808] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:48:04.850280] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:48:04.859915] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:48:09.886952] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:48:09.896861] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:48:14.918184] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:48:14.927733] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:48:19.954684] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:48:19.965812] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:48:24.996268] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:48:25.006323] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:48:30.029134] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:48:30.039524] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:48:35.065785] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:48:35.075568] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:48:40.102621] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:48:40.117494] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:48:45.141360] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:48:45.151444] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:48:50.174485] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:48:50.186053] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:48:55.206979] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:48:55.217845] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:00.238871] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:00.249824] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:05.276372] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:05.286180] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:10.311933] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:10.323008] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:15.348607] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:15.361991] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:20.387577] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:20.414214] end - ✅ in 0.026s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:25.436240] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:25.445885] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:30.469176] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:30.479056] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:35.502314] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:35.512219] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:40.534286] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:40.545153] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:45.570272] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:45.583225] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:50.612194] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:50.622266] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:49:55.646656] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:49:55.656930] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:00.684475] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:00.696161] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:05.717379] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:05.728218] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:10.753470] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:10.763313] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:15.786555] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:15.795778] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:20.821052] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:20.832855] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:25.860731] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:25.870635] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:30.892996] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:30.903123] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:35.933364] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:35.943528] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:40.966217] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:40.977102] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:46.000294] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:46.010848] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:51.036118] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:51.057824] end - ✅ in 0.021s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:50:56.081035] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:50:56.091489] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:01.111737] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:01.121275] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:06.150568] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:06.166598] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:11.190322] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:11.207480] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:16.235000] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:16.259261] end - ✅ in 0.024s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:21.287866] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:21.357525] end - ✅ in 0.069s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:26.385005] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:26.395474] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:31.424519] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:31.457035] end - ✅ in 0.032s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:36.532698] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:36.556732] end - ✅ in 0.023s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:41.584135] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:41.657762] end - ✅ in 0.073s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:46.689760] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:46.757774] end - ✅ in 0.068s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:51.798580] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:51.812701] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:51:56.836850] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:51:56.848474] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:01.872378] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:01.881599] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:06.994723] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:07.007571] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:12.031527] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:12.045030] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:17.068942] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:17.084527] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:22.111512] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:22.125149] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:27.148518] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:27.160025] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:32.189622] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:32.211503] end - ✅ in 0.021s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:37.237464] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:37.248001] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:42.379780] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:42.390230] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:47.418686] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:47.447981] end - ✅ in 0.029s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:52.476067] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:52.485678] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:52:57.504129] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:52:57.522158] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:02.586721] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:02.597366] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:07.624619] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:07.635482] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:12.657180] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:12.669856] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:17.691815] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:17.702932] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:22.728135] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:22.737490] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:27.758454] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:27.770024] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:32.796064] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:32.810039] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:37.842880] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:37.856255] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:42.881445] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:42.893212] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:47.921084] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:47.932353] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:52.954511] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:52.966309] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:53:57.991956] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:53:58.005298] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:03.031268] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:03.042308] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:08.074953] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:08.086135] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:13.109610] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:13.119871] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:18.145311] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:18.157217] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:23.180115] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:23.194708] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:28.218592] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:28.233318] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:33.254597] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:33.267440] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:38.295119] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:38.308147] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:43.330359] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:43.350602] end - ✅ in 0.020s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:48.372600] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:48.387066] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:53.410776] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:53.420794] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:54:58.439597] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:54:58.453046] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:03.480663] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:03.497692] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:08.525336] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:08.541682] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:13.571455] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:13.581269] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:18.601755] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:18.625221] end - ✅ in 0.023s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:23.653272] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:23.663354] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:28.685251] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:28.698801] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:33.725789] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:33.735723] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:38.759633] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:38.770073] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:43.796782] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:43.809724] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:48.835508] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:48.846179] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:53.869058] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:53.883502] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:55:58.906241] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:55:58.917702] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:03.941674] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:03.953878] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:08.979621] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:08.990083] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:14.016724] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:14.027657] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:19.052695] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:19.062387] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:24.088324] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:24.099048] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:29.120029] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:29.135705] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:34.159726] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:34.171235] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:39.196781] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:39.207014] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:44.231960] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:44.242868] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:49.268230] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:49.278462] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:54.304299] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:54.315002] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:56:59.341070] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:56:59.350977] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:04.372883] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:04.383179] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:09.404093] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:09.419229] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:14.448696] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:14.461158] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:19.487438] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:19.500703] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:24.524439] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:24.534862] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:29.558062] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:29.567970] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:34.596692] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:34.609142] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:39.631289] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:39.643059] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:44.667217] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:44.678230] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:49.701236] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:49.711146] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:54.748571] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:54.760846] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:57:59.786593] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:57:59.796277] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:58:04.821035] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:58:04.831368] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:58:09.860644] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:58:09.870535] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:58:14.894497] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:58:14.906102] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:58:19.925951] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:58:19.936051] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:58:24.961615] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:58:24.972543] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:58:29.998160] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:58:30.011070] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:58:35.035913] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:58:35.048766] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:58:40.073578] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:58:40.084571] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:58:45.108223] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:58:45.117609] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:58:50.141947] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:58:50.151910] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:58:55.178618] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:58:55.192471] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:00.213846] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:00.234918] end - ✅ in 0.021s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:05.258929] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:05.269779] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:10.291040] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:10.301784] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:15.323944] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:15.333840] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:20.359156] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:20.369742] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:25.395891] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:25.407236] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:30.428724] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:30.438721] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:35.466379] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:35.475931] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:40.498123] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:40.507771] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:45.529493] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:45.538980] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:50.562616] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:50.572957] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T17:59:55.598473] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T17:59:55.609601] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T18:00:00.633543] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T18:00:00.643835] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T18:00:05.670601] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T18:00:05.682163] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T18:00:10.709570] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T18:00:10.719323] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T18:00:15.745684] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T18:00:15.757026] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-16T18:00:20.779145] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-16T18:00:20.789599] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/fc-concurrency-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-07-16T18:00:20.816021] end - ❌ 902.132s: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_flow_control.py:169 [router-managed-scheduler-flow-control-concurrency-detector-workload-llmd-simulator] Failed: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-llmd-simulator2] ______ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='What is KServe?', service_name='llmisvc-rout... {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response from {service_name}") [e2e-llm-inference-service] > wait_for_model_response( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] test_case, [e2e-llm-inference-service] test_case.wait_timeout, [e2e-llm-inference-service] extra_headers=test_case.extra_headers, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:810: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'workload-llm... {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-07-16T17:51:48.058030', start_time = 1784224308.0586329 [e2e-llm-inference-service] duration = 903.6344721317291, timestamp_end = '2026-07-16T18:06:51.693106' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='What is KServe?', service_name='llmisvc-rout... {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] extra_headers = {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'} [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1119: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .get_successful_response at 0x7f137d4c4360> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Service returned 401: [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig router-managed-llmisvc-router-m-d2b356d9 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-router-m-d2b356d9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-router-m-d2b356d9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-53a6ad30 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-53a6ad30 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-53a6ad30 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-qw-a8136937 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-qw-a8136937 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-qw-a8136937 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-54bae102 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-54bae102 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-llmisvc-54bae102 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-21bdc061 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-21bdc061 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig model-qwen2-5-0-5b-llmisvc-mode-21bdc061 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-16T17:50:49.937948] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='What is KServe?', service_name='llmisvc-router-managed-test-llm-4b931143', endpoint='/v1/chat/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f137dd93100>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[TestCase(base_refs=['router-managed', 'workload-llmd-simulator', 'model-qwen2.5-0.5b'], prompt='What is KServe?', service_name='llmisvc-model-qwen2-5-0-5b-rout-4f8c0978', endpoint='/v1/chat/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f137dd931a0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/Qwen/Qwen2.5-0.5B-Instruct'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-qwen2-5-0-5b-rout-4f8c0978', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-qw-a8136937'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-54bae102'}, [e2e-llm-inference-service] {'name': 'model-qwen2-5-0-5b-llmisvc-mode-21bdc061'}]}, [e2e-llm-inference-service] 'status': None}, model_name='Qwen/Qwen2.5-0.5B-Instruct')], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-16T17:50:49.950966] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-16T17:50:50.071498] end - ✅ in 0.120s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-16T17:50:50.071649] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-16T17:51:16Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T17:51:16Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T17:51:16Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:51:16Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T17:51:16Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T17:51:16Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T17:51:16Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:51:16Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-16T17:51:43Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-router-managed-test-llm-4b931143-inference--ip-81883b73.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T17:51:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T17:51:43Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:51:16Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T17:51:43Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-router-managed-test-llm-4b931143-inference--ip-81883b73.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T17:51:43Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-router-managed-test-llm-4b931143-inference--ip-81883b73.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T17:51:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:51:43Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-16T17:51:46Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T17:51:43Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T17:51:43Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:51:16Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T17:51:46Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T17:51:46Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T17:51:43Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T17:51:43Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-07-16T17:51:48.057859] end - ✅ in 57.986s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-07-16T17:51:48.058030] start - args=(, TestCase(base_refs=['router-managed', 'workload-llmd-simulator'], prompt='What is KServe?', service_name='llmisvc-router-managed-test-llm-4b931143', endpoint='/v1/chat/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f137dd93100>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[TestCase(base_refs=['router-managed', 'workload-llmd-simulator', 'model-qwen2.5-0.5b'], prompt='What is KServe?', service_name='llmisvc-model-qwen2-5-0-5b-rout-4f8c0978', endpoint='/v1/chat/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f137dd931a0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/Qwen/Qwen2.5-0.5B-Instruct'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-qwen2-5-0-5b-rout-4f8c0978', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-qw-a8136937'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-54bae102'}, [e2e-llm-inference-service] {'name': 'model-qwen2-5-0-5b-llmisvc-mode-21bdc061'}]}, [e2e-llm-inference-service] 'status': None}, model_name='Qwen/Qwen2.5-0.5B-Instruct')], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:51:48.058641] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:51:48.069453] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:51:53.255300] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:51:53.269331] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:51:58.340187] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:51:58.350176] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:03.378213] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:03.388744] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:08.409041] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:08.431361] end - ✅ in 0.022s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:13.456818] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:13.471562] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:18.494711] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:18.509825] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:23.534903] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:23.547962] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:28.575243] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:28.587838] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:33.611861] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:33.622977] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:38.648724] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:38.659071] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:43.682176] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:43.697197] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:48.719046] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:48.733038] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:53.758140] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:53.769708] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:52:58.793145] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:52:58.806579] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:03.829060] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:03.845332] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:08.977053] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:08.988203] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:14.011561] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:14.023238] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:19.044376] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:19.054336] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:24.080527] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:24.090677] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:29.110480] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:29.126962] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:34.149926] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:34.168915] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:39.194588] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:39.206923] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:44.231272] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:44.247320] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:49.271802] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:49.284488] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 5: restarts={"openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-16T17:28:58Z\"}}"} resources={"httproutes": 4, "llminferenceservices": 4, "authpolicies": 2} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:54.304501] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:54.332559] end - ✅ in 0.028s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:53:59.357022] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:53:59.375390] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:04.397490] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:04.409020] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:09.434176] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:09.450248] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:14.470777] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:14.493858] end - ✅ in 0.023s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:19.519025] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:19.534338] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:24.559031] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:24.573292] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:29.599424] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:29.624695] end - ✅ in 0.025s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:34.646993] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:34.660188] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:39.682219] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:39.693166] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:44.718284] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:44.735857] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:49.762379] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:49.773149] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:54.798453] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:54.811955] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:54:59.836999] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:54:59.847921] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:55:04.872207] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:55:04.887977] end - ✅ in 0.015s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:55:09.912021] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:55:09.922163] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:55:14.946447] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:55:14.956348] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:55:19.980310] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:55:19.999832] end - ✅ in 0.019s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:55:25.022781] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:55:25.037065] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:55:30.060343] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:55:30.070425] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:55:35.094121] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:55:35.107944] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:55:40.139619] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:55:40.159783] end - ✅ in 0.020s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:55:45.182514] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:55:45.198970] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:55:50.225129] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:55:50.269145] end - ✅ in 0.044s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:55:55.294777] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:55:55.307519] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:00.332826] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:00.347663] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:05.371758] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:05.383017] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:10.405246] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:10.417447] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:15.444055] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:15.456945] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:20.479679] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:20.493197] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:25.518553] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:25.529681] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:30.552550] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:30.566384] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:35.591102] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:35.603390] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:40.627158] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:40.637624] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:45.658467] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:45.669333] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:50.694875] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:50.716437] end - ✅ in 0.021s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:56:55.738328] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:56:55.748843] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:00.768188] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:00.778633] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:05.803271] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:05.813365] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:10.837278] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:10.847480] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:15.871061] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:15.883148] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:20.903614] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:20.914723] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:25.940662] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:25.950987] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:30.974425] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:30.984566] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:36.008608] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:36.019998] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:41.045847] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:41.056215] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:46.078587] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:46.089007] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:51.114737] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:51.125459] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:57:56.149811] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:57:56.160171] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:01.187609] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:01.198782] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:06.219673] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:06.229632] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:11.251359] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:11.261972] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:16.282546] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:16.293182] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:21.319211] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:21.329832] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:26.350889] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:26.361675] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:31.387287] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:31.397844] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:36.425495] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:36.436488] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:41.503117] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:41.514112] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:46.536996] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:46.546526] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:51.571740] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:51.582031] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 6: restarts={"openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-16T17:28:58Z\"}}"} resources={"httproutes": 4, "llminferenceservices": 4, "authpolicies": 2} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:58:56.608446] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:58:56.619779] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:01.642037] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:01.659271] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:06.681496] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:06.693302] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:11.719348] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:11.730058] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:16.753108] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:16.763704] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:21.789921] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:21.800017] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:26.819436] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:26.831891] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:31.859460] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:31.869578] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:36.891628] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:36.902838] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:41.925414] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:41.936033] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:46.956927] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:46.966465] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:51.990912] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:52.001319] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T17:59:57.023647] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T17:59:57.035072] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:02.057536] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:02.067717] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:07.100099] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:07.111643] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:12.132477] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:12.142142] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:17.165665] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:17.178681] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:22.201866] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:22.257551] end - ✅ in 0.055s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:27.282666] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:27.357870] end - ✅ in 0.075s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:32.379089] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:32.462097] end - ✅ in 0.083s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:37.489167] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:37.658055] end - ✅ in 0.168s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:42.687151] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:42.858116] end - ✅ in 0.171s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:47.880646] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:47.958222] end - ✅ in 0.077s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:52.989636] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:53.257948] end - ✅ in 0.268s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:00:58.286236] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:00:58.299463] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:03.328089] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:03.339131] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:08.370596] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:08.457568] end - ✅ in 0.087s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:13.489220] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:13.557844] end - ✅ in 0.068s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:18.586329] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:18.658394] end - ✅ in 0.072s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:23.686154] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:23.696916] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:28.725384] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:28.757422] end - ✅ in 0.031s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:33.878083] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:33.957135] end - ✅ in 0.079s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:38.983280] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:38.992845] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:44.017934] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:44.057941] end - ✅ in 0.039s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:49.187619] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:49.196854] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:54.226974] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:54.237112] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:01:59.260575] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:01:59.271475] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:04.300636] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:04.310778] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:09.344525] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:09.355274] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:14.384470] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:14.393425] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:19.417589] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:19.427499] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:24.455186] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:24.465655] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:29.494195] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:29.506837] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:34.534414] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:34.548020] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:39.572835] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:39.582967] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:44.611551] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:44.621632] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:49.648197] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:49.657864] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:54.688743] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:54.700041] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:02:59.774962] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:02:59.784663] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:03:04.811264] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:03:04.821964] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:03:09.848343] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:03:09.858242] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:03:14.985233] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:03:14.995555] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:03:20.026511] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:03:20.037019] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:03:25.066391] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:03:25.084650] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:03:30.111127] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:03:30.121642] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:03:35.149181] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:03:35.159956] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:03:40.191256] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:03:40.201236] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:03:45.226577] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:03:45.238256] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:03:50.266946] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:03:50.279011] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:03:55.303559] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:03:55.315148] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 7: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "0\t{\"running\":{\"startedAt\":\"2026-07-16T18:01:10Z\"}}", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-16T17:28:58Z\"}}"} resources={"httproutes": 4, "llminferenceservices": 5, "authpolicies": 2} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:00.337093] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:00.347798] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:05.372078] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:05.383729] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:10.407888] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:10.417573] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:15.442591] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:15.456693] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:20.485277] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:20.495426] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:25.526392] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:25.536695] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:30.565166] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:30.575222] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:35.597696] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:35.608620] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:40.637740] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:40.648678] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:45.678365] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:45.688956] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:50.717354] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:50.727543] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:04:55.759553] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:04:55.769347] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:00.791780] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:00.802152] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:05.829048] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:05.839336] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:10.871113] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:10.881389] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:15.909181] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:15.921443] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:20.954804] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:20.965101] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:25.992984] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:26.005595] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:31.028368] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:31.038482] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:36.067852] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:36.077892] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:41.104598] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:41.115197] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:46.139793] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:46.150188] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:51.174387] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:51.184370] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:05:56.214248] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:05:56.229261] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:06:01.256591] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:06:01.266626] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:06:06.291049] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:06:06.301172] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:06:11.331225] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:06:11.342262] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:06:16.370602] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:06:16.383982] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:06:21.410201] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:06:21.420717] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:06:26.455636] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:06:26.467264] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:06:31.497491] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:06:31.508562] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:06:36.536980] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:06:36.546901] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:06:41.574158] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:06:41.583976] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:06:46.607675] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:06:46.618712] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:06:51.649542] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-router-managed-test-llm-4b931143', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-router-m-d2b356d9'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-llmisvc-53a6ad30'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-router-managed-test-llm-4b931143: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:06:51.661677] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/chat/completions with payload {'model': 'facebook/opt-125m', 'messages': [{'role': 'user', 'content': 'What is KServe?'}], 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-07-16T18:06:51.693106] end - ❌ 903.634s: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:825 [router-managed-workload-llmd-simulator] ❌ ERROR: Failed to call llm inference service llmisvc-router-managed-test-llm-4b931143: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-router-managed-test-llm-4b931143' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:50:50Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:50:49Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:50:50Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:51:47Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '38228' [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-router-m-d2b356d9 [e2e-llm-inference-service] - name: workload-llmd-simulator-llmisvc-53a6ad30 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:47Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:43Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:43Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:16Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:47Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:47Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:47Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:43Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-66746bd8b9-f297r to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: autoscale-hpa-deploy-kserve-66746bd8b9-f297r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set autoscale-hpa-deploy-kserve-66746bd8b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/autoscale-hpa-deploy-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:02:04 kserve-ci-e2e-test LLMInferenceServiceController Warning ScalingCRDNotFound Required scaling CRD not installed: failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:54 kserve-ci-e2e-test LLMInferenceServiceController Warning Error Reconciliation failed: failed to reconcile workload: failed to reconcile scaling: failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-concurrency-test-kserve-b5d44467c-xdjqc to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-concurrency-test-kserve-b5d44467c-xdjqc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-concurrency-test-kserve-router-scheduler-78c47c8645 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-concurrency-test-kserve-b5d44467c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/fc-concurrency-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-concurrency-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-concurrency-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/fc-concurrency-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-fc-concurrency-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/fc-concurrency-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/fc-concurrency-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-concurrency-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-concurrency-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/fc-concurrency-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:45:18 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-concurrency-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/fc-concurrency-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/fc-concurrency-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:45:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [fc-concurrency-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:25 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-concurrency-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-smoke-test-kserve-7f4687897-r5tjx to ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" in 1.6s (1.6s including waiting). Image size: 79178094 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-smoke-test-kserve-7f4687897-r5tjx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.25/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:02 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" in 1.538s (1.538s including waiting). Image size: 179512283 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-smoke-test-kserve-router-scheduler-fcbfbd8bf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-smoke-test-kserve-7f4687897 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/fc-smoke-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-smoke-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-smoke-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/fc-smoke-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-fc-smoke-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/fc-smoke-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/fc-smoke-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-smoke-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-smoke-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/fc-smoke-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:36 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with scheduler-v06-pd-migration-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/fc-smoke-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/fc-smoke-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [fc-smoke-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:57 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with scheduler-v06-threshold-migration-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with precise-prefix-cache-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:13 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with llmisvc-router-managed-test-llm-e45d1f79 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:15 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:50 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:25 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xj8kf to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:45 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:45 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:46 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.31:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xj8kf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:32 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "gw-section-name-router-with-gat-2f0a622e": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "gw-section-name-router-with-gat-2f0a622e": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj4qcm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj4qcm to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:06 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:06 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:06 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedMount MountVolume.SetUp failed for volume "tls-certs" : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs to ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:10 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:24 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:47 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2 to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:08 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:45 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:45 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:49 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:24 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-router-managed-test-llm-4b931143 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-sskh8 to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:44 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:44 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-sskh8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:05 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:43 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:43 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexb8dr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5657fd78d9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexb8dr to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:44 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-6f9bc95cfd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:14 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:14 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:14 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-69b67488bb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:08 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/model-fb-opt-125m-gw-section-na-8bf35153: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:35 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/model-fb-opt-125m-gw-section-na-e3d39686: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d6c8b5f7c-kgjrz to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" in 4.327s (4.327s including waiting). Image size: 300595129 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:02 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:02 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:02 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d6c8b5f7c-nmsvz to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6d6c8b5f7c-nmsvz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6d6c8b5f7c-kgjrz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-75779d4bffhzf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75779d4bffhzf to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" in 4.016s (4.016s including waiting). Image size: 300595129 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:04 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:05 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:05 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:59 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/llm-d-kv-cache:v0.8.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:43 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/llm-d-kv-cache:v0.8.0" in 43.867s (43.867s including waiting). Image size: 2935159967 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:43 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:43 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.54:8082/healthz": dial tcp 10.133.0.54:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:04 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning BackOff Back-off restarting failed container main in pod precise-prefix-cache-test-kserve-router-scheduler-75779d4bffhzf_kserve-ci-e2e-test(9ff96acc-75a8-48e2-9709-8abb012acf3f) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:15 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.133.0.54:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: Get "http://10.133.0.54:8082/healthz": dial tcp 10.133.0.54:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:48 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-75779d4b5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-6d6c8b5f7c from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:39 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/precise-prefix-cache-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:38 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kfbtn to ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:10 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:10 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:10 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:16 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.32:15021/healthz/ready": dial tcp 10.134.0.32:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-qbmpr to ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-qbmpr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-qbmpr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-kfbtn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gx98l to ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:41 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:41 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:41 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:39 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Warning Unhealthy Readiness probe failed: Get "http://10.134.0.30:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:46 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.30:15021/healthz/ready": dial tcp 10.134.0.30:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Warning Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-gx98l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-gx98l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:21 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:24 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-54c47cdfc7 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-799f46c59b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:34 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:07 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/router-with-gateway-ref-gw-sect-bd0c4d94: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:33 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/router-with-gateway-section-nam-5d237677: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:34 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/router-with-managed-route-gw-se-010f4010: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:08 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/router-with-managed-route-gw-se-bc1f68e4: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-68f9854ff-82kzx to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.24/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" in 2.361s (2.361s including waiting). Image size: 166803393 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:51 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" in 1.483s (1.483s including waiting). Image size: 79178094 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-68f9854ff-82kzx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-4cx7q to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:49 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" in 1.754s (1.754s including waiting). Image size: 79178094 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:49 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:49 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.49:8000/health": dial tcp 10.133.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-4cx7q [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-router-scheduler-79ww96l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-79ww96l to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:48 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" in 2.55s (2.55s including waiting). Image size: 179512283 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:42 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-router-scheduler-79c5847dfd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-68f9854ff from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-pd-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-pd-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:29 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/scheduler-v06-pd-migration-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:28 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-v06-pd-migration-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5d54f65f57-27ctz to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.26/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-5d54f65f57-27ctz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-prefill-b58fmzjxs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-b58fmzjxs to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-prefill-b58fbd55d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6ws58 to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedToRetrieveImagePullSecret Unable to retrieve some image pull secrets (scheduler-v06-threshold-migration-test-epp-sa-dockercfg-klsww); attempting to pull the image may not succeed. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:38 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-router-sched6ws58 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-router-scheduler-5db9bd4595 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-5d54f65f57 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-threshold-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-threshold-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:29 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/scheduler-v06-threshold-migration-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:29 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-v06-threshold-migration-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:35 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/workload-llmd-simulator-gw-sect-0fb35c52: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:09 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/workload-llmd-simulator-gw-sect-81946990: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 I0716 17:51:13.966591 1 config.go:808] "Configuration:" =< [e2e-llm-inference-service] { [e2e-llm-inference-service] "DisplayModelName": "facebook/opt-125m", [e2e-llm-inference-service] "IP": "", [e2e-llm-inference-service] "PodName": "", [e2e-llm-inference-service] "PodNameSpace": "", [e2e-llm-inference-service] "VllmDevMode": false, [e2e-llm-inference-service] "block-size": 16, [e2e-llm-inference-service] "data-parallel-rank": -1, [e2e-llm-inference-service] "data-parallel-size": 1, [e2e-llm-inference-service] "dataset-in-memory": false, [e2e-llm-inference-service] "dataset-path": "", [e2e-llm-inference-service] "dataset-table-name": "llmd", [e2e-llm-inference-service] "dataset-url": "", [e2e-llm-inference-service] "default-embedding-dimensions": 384, [e2e-llm-inference-service] "ec-transfer-config": "", [e2e-llm-inference-service] "enable-kvcache": false, [e2e-llm-inference-service] "enable-prefix-caching": false, [e2e-llm-inference-service] "enable-request-id-headers": false, [e2e-llm-inference-service] "enable-sleep-mode": false, [e2e-llm-inference-service] "enforce-eager": false, [e2e-llm-inference-service] "event-batch-size": 16, [e2e-llm-inference-service] "failure-injection-rate": 0, [e2e-llm-inference-service] "failure-types": null, [e2e-llm-inference-service] "fake-metrics": null, [e2e-llm-inference-service] "fake-metrics-refresh-interval": "100ms", [e2e-llm-inference-service] "force-dummy-tokenizer": true, [e2e-llm-inference-service] "global-cache-hit-threshold": 0, [e2e-llm-inference-service] "hash-seed": "", [e2e-llm-inference-service] "image-emission-rate": 0, [e2e-llm-inference-service] "inter-token-latency": "0s", [e2e-llm-inference-service] "inter-token-latency-std-dev": "0s", [e2e-llm-inference-service] "kv-cache-size": 1024, [e2e-llm-inference-service] "kv-cache-transfer-latency": "0s", [e2e-llm-inference-service] "kv-cache-transfer-latency-std-dev": "0s", [e2e-llm-inference-service] "kv-cache-transfer-time-per-token": "0s", [e2e-llm-inference-service] "kv-cache-transfer-time-std-dev": "0s", [e2e-llm-inference-service] "latency-calculator": "", [e2e-llm-inference-service] "log-http": false, [e2e-llm-inference-service] "lora-modules": null, [e2e-llm-inference-service] "max-cpu-loras": 1, [e2e-llm-inference-service] "max-loras": 1, [e2e-llm-inference-service] "max-model-len": 1024, [e2e-llm-inference-service] "max-num-seqs": 5, [e2e-llm-inference-service] "max-request-body-size-mb": 4, [e2e-llm-inference-service] "max-tool-call-array-param-length": 5, [e2e-llm-inference-service] "max-tool-call-integer-param": 100, [e2e-llm-inference-service] "max-tool-call-number-param": 100, [e2e-llm-inference-service] "max-waiting-queue-length": 1000, [e2e-llm-inference-service] "min-tool-call-array-param-length": 1, [e2e-llm-inference-service] "min-tool-call-integer-param": 0, [e2e-llm-inference-service] "min-tool-call-number-param": 0, [e2e-llm-inference-service] "mm-encoder-only": false, [e2e-llm-inference-service] "mm-processor-kwargs": "", [e2e-llm-inference-service] "mm-render-timeout": "1m0s", [e2e-llm-inference-service] "mode": "random", [e2e-llm-inference-service] "model": "facebook/opt-125m", [e2e-llm-inference-service] "object-tool-call-not-required-field-probability": 50, [e2e-llm-inference-service] "omni": false, [e2e-llm-inference-service] "port": 8000, [e2e-llm-inference-service] "prefill-overhead": "0s", [e2e-llm-inference-service] "prefill-time-per-token": "0s", [e2e-llm-inference-service] "prefill-time-std-dev": "0s", [e2e-llm-inference-service] "render-timeout": "30s", [e2e-llm-inference-service] "render-url": "http://localhost:8082", [e2e-llm-inference-service] "seed": 1784224273966104800, [e2e-llm-inference-service] "self-signed-certs": false, [e2e-llm-inference-service] "served-model-name": [ [e2e-llm-inference-service] "facebook/opt-125m" [e2e-llm-inference-service] ], [e2e-llm-inference-service] "ssl-certfile": "/var/run/kserve/tls/tls.crt", [e2e-llm-inference-service] "ssl-keyfile": "/var/run/kserve/tls/tls.key", [e2e-llm-inference-service] "startup-duration": "0s", [e2e-llm-inference-service] "time-factor-under-load": 1, [e2e-llm-inference-service] "time-to-first-token": "0s", [e2e-llm-inference-service] "time-to-first-token-std-dev": "0s", [e2e-llm-inference-service] "time-to-generate-image": "0s", [e2e-llm-inference-service] "time-to-generate-image-std-dev": "0s", [e2e-llm-inference-service] "tool-call-not-required-param-probability": 50, [e2e-llm-inference-service] "use-vllm-map-event-format": false, [e2e-llm-inference-service] "zmq-endpoint": "tcp://127.0.0.1:5557" [e2e-llm-inference-service] } [e2e-llm-inference-service] > [e2e-llm-inference-service] I0716 17:51:13.966672 1 tokenizer.go:60] "Force dummy tokenizer flag is set, using simulated tokenizer" model="facebook/opt-125m" [e2e-llm-inference-service] I0716 17:51:13.971235 1 context.go:216] "No dataset path or URL provided, using random text for responses" [e2e-llm-inference-service] I0716 17:51:13.971419 1 communication.go:66] "Starting communication layer" [e2e-llm-inference-service] I0716 17:51:13.971742 1 http_server_tls.go:59] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key" [e2e-llm-inference-service] I0716 17:51:13.972204 1 simulator.go:210] "Start processing routine" [e2e-llm-inference-service] I0716 17:51:13.972243 1 grpc.go:129] "Server starting" protocol="gRPC" port=8000 [e2e-llm-inference-service] I0716 17:51:13.972680 1 http.go:117] "Server starting" protocol="HTTPS" port=8000 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1784224274.2808301,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1784224274.2809572,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n weight: 2\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-router-managed-test-llm-4b931143-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1784224274.2810888,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.2815742,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.2816198,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: prefix-cache-scorer} {Type: no-hit-lru-scorer} {Type: max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: no-hit-lru-scorer, Weight: 2.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.2816308,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.2819936,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"no-hit-lru-scorer","parameters":null,"type":"no-hit-lru-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"no-hit-lru-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3001385,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1784224274.300213,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3002462,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.300282,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.30034,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3003592,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3004277,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: no-hit-lru-scorer, Type: no-hit-lru-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: no-hit-lru-scorer, Weight: 2.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3004932,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3005493,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3005672,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3005903,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3006637,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3006723,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc00032d740 DataConfig:{Sources:[{Plugin:0xc0002a0240 Extractors:[0xc00032d940]}]} FlowControlConfig: ParserRegistry:0xc00032df00}"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.316766,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3167942,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3168085,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3168132,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.316818,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3168225,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3168263,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3168306,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3168344,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.31684,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3168442,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3168488,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.316856,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3168607,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.316883,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1784224274.316891,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3169038,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3169165,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3169308,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3170009,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1784224274.317015,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3170466,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3171916,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3174744,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1784224274.317466,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3176,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3175952,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3177197,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3177447,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3183703,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3184543,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1784224274.323268,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3244507,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3247576,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.3249638,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.418883,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.4189105,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784224274.418883,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.4189293,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784224274.5190353,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.5190382,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1784224274.519067,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784224274.5190747,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784224274.5192044,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-router-managed-test-llm-4b931143-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-4b931143-inference-pool","reconcileID":"4d5ef312-34ad-4628-9c32-9cd05d816131"} [e2e-llm-inference-service] {"level":"info","ts":1784224283.86414,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs","reconcileID":"f792b5b6-564e-41a4-97a2-2af729e382ae"} [e2e-llm-inference-service] {"level":"info","ts":1784224301.845892,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-router-managed-test-llm-4b931143-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-router-managed-test-llm-4b931143-inference-pool","reconcileID":"45e1f787-c661-4bc1-afbb-58c499f06c80"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 05b15cca-9925-4c75-9bb1-89a0731ea52e [e2e-llm-inference-service] resourceVersion: '38209' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.58 [e2e-llm-inference-service] nodeName: ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf [e2e-llm-inference-service] uid: 1500b232-42ed-4480-a44b-41139e735a71 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6e717d1c-cb03-4e1b-9a8b-30494b42254e [e2e-llm-inference-service] resourceVersion: '37918' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.29 [e2e-llm-inference-service] nodeName: ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs [e2e-llm-inference-service] uid: 96495a17-94db-46ed-8b38-53c7c97e598e [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 96495a17-94db-46ed-8b38-53c7c97e598e [e2e-llm-inference-service] resourceVersion: '37917' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 85bf867969 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.29/23"],"mac_address":"0a:58:0a:86:00:1d","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.29/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.29\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:1d\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969 [e2e-llm-inference-service] uid: 7fe54e75-af03-4bcc-8b8c-5cedb7ec6484 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-133-160 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7fe54e75-af03-4bcc-8b8c-5cedb7ec6484"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.29"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-mg2hh [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-mg2hh [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] hostIP: 10.0.133.160 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.133.160 [e2e-llm-inference-service] podIP: 10.134.0.29 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.29 [e2e-llm-inference-service] startTime: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim-dev@sha256:151063768d4a46d17a30ff512d74946b20cf653a2302601a6a823727fbbe4026 [e2e-llm-inference-service] containerID: cri-o://81d0a7e4bf4060017f31b266e9c26714d6d1d4890c77fdc94bca19ae516d049a [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-mg2hh [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf [e2e-llm-inference-service] generateName: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-6f9bc95cfd- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1500b232-42ed-4480-a44b-41139e735a71 [e2e-llm-inference-service] resourceVersion: '38208' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6f9bc95cfd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.58/23"],"mac_address":"0a:58:0a:85:00:3a","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.58/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.58\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:3a\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-6f9bc95cfd [e2e-llm-inference-service] uid: f1463a70-a13e-4c2e-a642-44f755a3b2b1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-132-130 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f1463a70-a13e-4c2e-a642-44f755a3b2b1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.58"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-lkhgp [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-lkhgp [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-2f9wr [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:15Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] hostIP: 10.0.132.130 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.132.130 [e2e-llm-inference-service] podIP: 10.133.0.58 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.58 [e2e-llm-inference-service] startTime: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://e4ba736399112105c63519fcd5f00c37ce8b319b6f8255cc4f33f62a2a92ab86 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-lkhgp [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 55024758-59f1-478d-9b2d-f02159b67c20 [e2e-llm-inference-service] resourceVersion: '37644' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-2f9wr [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-2f9wr"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-sdm8r"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-2f9wr [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] - name: llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-2f9wr [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b2d24835-51a5-4694-902b-3807f9c44324 [e2e-llm-inference-service] resourceVersion: '37671' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.32.183 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.32.183 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: acda0a81-5d31-44ba-bca9-22497cdccd29 [e2e-llm-inference-service] resourceVersion: '37626' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.33.58 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.33.58 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 54000681-92f1-4b49-a077-58904cd44642 [e2e-llm-inference-service] resourceVersion: '37921' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e038d019-37db-4376-9007-2a2f7bf5e31e [e2e-llm-inference-service] resourceVersion: '38213' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n- type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ no-hit-lru-scorer\n weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-6f9bc95cfd" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7fe54e75-af03-4bcc-8b8c-5cedb7ec6484 [e2e-llm-inference-service] resourceVersion: '37920' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 85bf867969 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] uid: 54000681-92f1-4b49-a077-58904cd44642 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"54000681-92f1-4b49-a077-58904cd44642"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 85bf867969 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 85bf867969 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-6f9bc95cfd [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f1463a70-a13e-4c2e-a642-44f755a3b2b1 [e2e-llm-inference-service] resourceVersion: '38211' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6f9bc95cfd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] uid: e038d019-37db-4376-9007-2a2f7bf5e31e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e038d019-37db-4376-9007-2a2f7bf5e31e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6f9bc95cfd [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6f9bc95cfd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n- type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ no-hit-lru-scorer\n weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 24a195d6-c15f-48a8-b27d-ecc76443e18a [e2e-llm-inference-service] resourceVersion: '37652' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6e768718-a106-4d34-93a2-f0fd889a43bb [e2e-llm-inference-service] resourceVersion: '37650' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service-z6qph [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-4b931143-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c4883e44-0507-4174-9873-e94568266068 [e2e-llm-inference-service] resourceVersion: '38210' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] uid: b2d24835-51a5-4694-902b-3807f9c44324 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b2d24835-51a5-4694-902b-3807f9c44324"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.58 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf [e2e-llm-inference-service] uid: 1500b232-42ed-4480-a44b-41139e735a71 [e2e-llm-inference-service] nodeName: ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-spcxs7 [e2e-llm-inference-service] generateName: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2d0a32ce-e66a-4ac1-bad0-c51714e0a55e [e2e-llm-inference-service] resourceVersion: '37919' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] uid: acda0a81-5d31-44ba-bca9-22497cdccd29 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:23Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"acda0a81-5d31-44ba-bca9-22497cdccd29"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.29 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs [e2e-llm-inference-service] uid: 96495a17-94db-46ed-8b38-53c7c97e598e [e2e-llm-inference-service] nodeName: ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 24a195d6-c15f-48a8-b27d-ecc76443e18a [e2e-llm-inference-service] resourceVersion: '37652' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6e768718-a106-4d34-93a2-f0fd889a43bb [e2e-llm-inference-service] resourceVersion: '37650' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:41Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:51:42Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:51:43Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '38171' [e2e-llm-inference-service] uid: 9cd0765a-d62c-4a1f-80df-5904ac7be08a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:42Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:15Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:41Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:51:42Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:51:43Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '38171' [e2e-llm-inference-service] uid: 9cd0765a-d62c-4a1f-80df-5904ac7be08a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:42Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:15Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:51:41Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '38139' [e2e-llm-inference-service] uid: 5ee74dc4-ba31-4ff5-8c92-476463c70b11 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:41Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:41Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:16Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:16Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T17:51:20Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '37872' [e2e-llm-inference-service] uid: cbbdeba2-7c6e-44f4-8645-9202379b7e29 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:18Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T17:51:20Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '37699' [e2e-llm-inference-service] uid: 691aa793-1bf7-428a-8712-01d5d95d725f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:41Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:41Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '38145' [e2e-llm-inference-service] uid: d4312e7c-43b9-449b-aee7-231d3de2204f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-inference--ip-81883b73.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '37709' [e2e-llm-inference-service] uid: a73440a9-37ce-466e-9c00-e03f1a3b8de9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '37699' [e2e-llm-inference-service] uid: 691aa793-1bf7-428a-8712-01d5d95d725f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:41Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:41Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '38145' [e2e-llm-inference-service] uid: d4312e7c-43b9-449b-aee7-231d3de2204f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-inference--ip-81883b73.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '37709' [e2e-llm-inference-service] uid: a73440a9-37ce-466e-9c00-e03f1a3b8de9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '37699' [e2e-llm-inference-service] uid: 691aa793-1bf7-428a-8712-01d5d95d725f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:41Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:41Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '38145' [e2e-llm-inference-service] uid: d4312e7c-43b9-449b-aee7-231d3de2204f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-inference--ip-81883b73.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:14Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '37709' [e2e-llm-inference-service] uid: a73440a9-37ce-466e-9c00-e03f1a3b8de9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4ef86008-3839-4d2c-9074-3c6019f31537"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T17:51:13Z' [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] uid: 4ef86008-3839-4d2c-9074-3c6019f31537 [e2e-llm-inference-service] resourceVersion: '37679' [e2e-llm-inference-service] uid: 95644d5d-1746-42e5-82b5-86736900a0af [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:06:53Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 85bf867969 [e2e-llm-inference-service] timestamp: '2026-07-16T18:06:47Z' [e2e-llm-inference-service] window: 14.416s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 17201234n [e2e-llm-inference-service] memory: 28140Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:06:53Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-router-managed-test-llm-4b931143 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6f9bc95cfd [e2e-llm-inference-service] timestamp: '2026-07-16T18:06:44Z' [e2e-llm-inference-service] window: 12.441s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 9358170n [e2e-llm-inference-service] memory: 30384Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-16T18:06:53.560266] end - ❌ 963.622s: Service returned 401: [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_deployment(test_case: TestCase): [e2e-llm-inference-service] """HPA + Deployment: VA and HPA exist; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:537: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:478: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-16T18:01:10.556646', start_time = 1784224870.5569367 [e2e-llm-inference-service] duration = 900.5954418182373, timestamp_end = '2026-07-16T18:16:11.152420' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] ...toscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f02eae36d40> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'status': 'Unknown', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1199: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:141 Patched Gateway kserve-ci-e2e-test/router-gateway-1 with parametersRef [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:163 Gateway kserve-ci-e2e-test/router-gateway-1 is Programmed [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_deployment] [2026-07-16T18:01:09.717041] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-16T18:01:09.729737] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-16T18:01:10.556484] end - ✅ in 0.827s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-16T18:01:10.556646] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'status': 'Unknown', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'status': 'Unknown', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-16T18:16:11.152420] end - ❌ 900.595s: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'status': 'Unknown', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-16T18:16:11.152634] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: 1 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-66746bd8b9-f297r'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-66746bd8b9-f297r'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-16T18:16:16.231206] end - ✅ in 5.078s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_deployment] [2026-07-16T18:16:16.231345] end - ❌ 906.514s: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'status': 'Unknown', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T18:01:23Z', 'message': 'failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1"', 'reason': 'ScalingCRDNotFound', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_n... {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response from {service_name}") [e2e-llm-inference-service] > wait_for_model_response( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] test_case, [e2e-llm-inference-service] test_case.wait_timeout, [e2e-llm-inference-service] extra_headers=test_case.extra_headers, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:810: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'workload-sin... {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-07-16T18:11:17.474076', start_time = 1784225477.4744217 [e2e-llm-inference-service] duration = 900.2016968727112, timestamp_end = '2026-07-16T18:26:17.676119' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_n... {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] extra_headers = {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'} [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1119: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .get_successful_response at 0x7f137d4c4ae0> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Service returned 401: [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-fb-66e80b02 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-fb-66e80b02 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-fb-66e80b02 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-1ae2b31a in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-1ae2b31a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-1ae2b31a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1761 Checking LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-c0d503b0 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1787 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-c0d503b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1797 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-c0d503b0 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-16T18:09:52.305807] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_name='llmisvc-model-fb-opt-125m-with-ba4d693a', endpoint='/v1/models', max_tokens=20, payload_formatter=None, response_assertion=.response_assertion at 0x7f137dd932e0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1812 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-16T18:09:52.322774] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-16T18:09:52.469477] end - ✅ in 0.146s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-16T18:09:52.469638] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-16T18:09:59Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T18:09:59Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T18:09:59Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T18:09:59Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T18:09:59Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T18:09:59Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T18:09:59Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T18:09:59Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-16T18:10:29Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T18:10:29Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T18:10:29Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T18:09:59Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T18:10:29Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T18:10:29Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T18:10:29Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T18:10:29Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-16T18:10:33Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-16T18:10:33Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-16T18:10:33Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-16T18:09:59Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-16T18:10:33Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-16T18:10:33Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-16T18:10:33Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-16T18:10:33Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-07-16T18:11:17.473923] end - ✅ in 85.004s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-07-16T18:11:17.474076] start - args=(, TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_name='llmisvc-model-fb-opt-125m-with-ba4d693a', endpoint='/v1/models', max_tokens=20, payload_formatter=None, response_assertion=.response_assertion at 0x7f137dd932e0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:11:17.474429] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:11:17.557673] end - ✅ in 0.083s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:11:22.591479] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:11:22.601765] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:11:27.632216] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:11:27.643278] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:11:32.674728] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:11:32.685822] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:11:37.784462] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:11:37.796183] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:11:42.825707] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:11:42.836437] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:11:47.860544] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:11:47.871095] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:11:52.893390] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:11:52.904067] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:11:57.933750] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:11:57.943314] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:02.975218] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:02.985284] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:08.017012] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:08.033718] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:13.089185] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:13.099356] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:18.131065] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:18.141030] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:23.167507] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:23.178354] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:28.209581] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:28.220249] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:33.247258] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:33.257774] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:38.286120] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:38.298082] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:43.330464] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:43.341583] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:48.368186] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:48.377914] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:53.405971] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:53.416918] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:12:58.446033] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:12:58.457339] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:03.490373] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:03.501800] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:08.730956] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:08.741514] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:13.767880] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:13.778230] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:18.808235] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:18.818227] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:23.851932] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:23.863027] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:28.894933] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:28.904996] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:33.934380] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:33.945256] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:38.970362] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:38.980759] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:44.008957] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:44.020172] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:49.048063] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:49.058759] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:54.087542] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:54.098187] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:13:59.144770] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:13:59.159187] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:04.189345] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:04.199359] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:09.252323] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:09.262529] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 9: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "0\t{\"running\":{\"startedAt\":\"2026-07-16T18:01:10Z\"}}", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-16T17:28:58Z\"}}"} resources={"httproutes": 5, "llminferenceservices": 6, "authpolicies": 3} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:14.286139] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:14.299065] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:19.319521] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:19.329248] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:24.351555] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:24.363633] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:29.392943] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:29.402837] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:34.430714] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:34.440708] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:39.462205] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:39.472796] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:44.500466] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:44.517190] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:49.543022] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:49.553707] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:54.584124] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:54.595699] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:14:59.625252] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:14:59.636643] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:15:04.793127] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:15:04.803487] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:15:09.831607] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:15:09.841858] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:15:14.868134] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:15:14.878185] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 500: Internal Server Error. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Service returned 500: Internal Server Error. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:15:20.127599] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:15:20.138224] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:15:25.165732] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:15:25.176095] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:15:30.280538] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:15:30.290542] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:15:35.316977] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:15:35.327635] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:15:40.531007] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:15:40.548792] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:15:45.580260] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:15:45.591242] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:15:50.690179] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:15:50.701874] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:15:55.732649] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:15:55.743448] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:00.779384] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:00.790889] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:05.896071] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:05.907960] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:10.936062] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:10.946500] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:15.975845] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:15.986341] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:21.015779] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:21.026307] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:26.055420] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:26.066082] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:31.093363] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:31.103739] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:36.135705] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:36.145783] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:41.179192] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:41.191001] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:46.290986] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:46.301849] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:51.331160] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:51.341221] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:16:56.529552] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:16:56.539991] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:01.568991] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:01.579889] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:06.604814] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:06.615749] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:11.647139] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:11.657030] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:16.684331] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:16.695237] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:21.722823] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:21.734660] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:26.768571] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:26.778088] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:31.809343] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:31.819304] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:36.851373] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:36.862174] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:41.886292] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:41.896259] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:46.922305] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:46.932808] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:52.094187] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:52.104754] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:17:57.139857] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:17:57.153610] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:02.194616] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:02.204970] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:07.292469] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:07.303952] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:12.329939] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:12.339798] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:17.455148] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:17.465540] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:22.497585] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:22.508338] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:27.588871] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:27.598870] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:32.636728] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:32.652857] end - ✅ in 0.016s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:37.683444] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:37.697658] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:42.724839] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:42.735076] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:47.765286] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:47.775597] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:52.810584] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:52.821940] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:18:57.850477] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:18:57.861023] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:02.891077] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:02.901539] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:08.133803] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:08.144718] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:13.172292] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:13.182545] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 10: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "0\t{\"running\":{\"startedAt\":\"2026-07-16T18:01:10Z\"}}", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-16T17:28:58Z\"}}"} resources={"httproutes": 5, "llminferenceservices": 5, "authpolicies": 3} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:18.211102] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:18.229477] end - ✅ in 0.018s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:23.257438] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:23.269355] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:28.294709] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:28.304873] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:33.336085] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:33.346906] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:38.382680] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:38.392724] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:43.426495] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:43.437095] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:48.535575] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:48.546685] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:53.573433] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:53.583319] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 500: Internal Server Error. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Service returned 500: Internal Server Error. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:19:58.827287] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:19:58.837760] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:03.864925] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:03.877045] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:08.910421] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:08.920372] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:13.945886] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:13.956807] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:18.986479] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:18.997095] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:24.030445] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:24.040593] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:29.070648] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:29.091622] end - ✅ in 0.020s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:34.141474] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:34.151378] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:39.180624] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:39.191191] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:44.221703] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:44.232746] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:49.259935] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:49.270684] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:54.303019] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:54.314226] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:20:59.345878] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:20:59.356523] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:04.387602] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:04.397629] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:09.430180] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:09.440915] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:14.472137] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:14.481825] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:19.510176] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:19.520822] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:24.552099] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:24.563513] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:29.589303] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:29.599381] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:34.629622] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:34.639845] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:39.672927] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:39.683935] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:44.715069] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:44.725205] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:49.751909] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:49.761957] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:54.792990] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:54.805988] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:21:59.840129] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:21:59.851133] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:22:04.881074] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:22:04.891365] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:22:09.994345] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:22:10.004572] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:22:15.039240] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:22:15.049573] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:22:20.190653] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:22:20.200916] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:22:25.230833] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:22:25.240700] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:22:30.446340] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:22:30.456997] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:22:35.485629] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:22:35.496244] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:22:40.525085] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:22:40.535712] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:22:45.562937] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:22:45.573796] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:22:50.604418] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:22:50.614006] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:22:55.653831] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:22:55.664027] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:00.697259] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:00.707586] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:05.734364] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:05.744035] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:10.774390] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:10.784228] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:16.047820] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:16.057993] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:21.087670] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:21.097667] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:26.125977] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:26.135991] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:31.167672] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:31.178319] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:36.206873] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:36.217840] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:41.248206] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:41.261248] end - ✅ in 0.012s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:46.294020] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:46.304624] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:51.327579] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:51.337393] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:23:56.371111] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:23:56.388488] end - ✅ in 0.017s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:01.432756] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:01.444095] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:06.476801] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:06.488206] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:11.526710] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:11.536292] end - ✅ in 0.009s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:16.566082] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:16.576921] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 11: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "0\t{\"running\":{\"startedAt\":\"2026-07-16T18:01:10Z\"}}", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-16T17:28:58Z\"}}"} resources={"httproutes": 5, "llminferenceservices": 5, "authpolicies": 3} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:21.679881] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:21.689901] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:26.717351] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:26.727304] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:31.754775] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:31.765070] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:36.792538] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:36.806364] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:41.833838] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:41.845499] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:46.880159] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:46.891170] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:51.919583] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:51.933224] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:24:56.960485] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:24:56.971120] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:01.999698] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:02.010275] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:07.040952] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:07.054029] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:12.085978] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:12.096057] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:17.125864] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:17.158600] end - ✅ in 0.032s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:22.185637] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:22.196059] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:27.237331] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:27.247955] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:32.282172] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:32.292245] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:37.319025] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:37.329864] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:42.358127] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:42.368261] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:47.399535] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:47.410603] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:52.440797] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:52.451555] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:25:57.481748] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:25:57.492848] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:26:02.516987] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:26:02.526912] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:26:07.556248] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:26:07.567793] end - ✅ in 0.011s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:26:12.597727] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:26:12.607908] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-16T18:26:17.635867] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:176 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-16T18:26:17.646760] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-07-16T18:26:17.676119] end - ❌ 900.202s: Service returned 401: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:825 [router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf] ❌ ERROR: Failed to call llm inference service llmisvc-model-fb-opt-125m-with-ba4d693a: Service returned 401: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-fb-opt-125m-with-ba4d693a' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:52Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:09:52Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:09:52Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '52199' [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-fb-66e80b02 [e2e-llm-inference-service] - name: workload-single-cpu-llmisvc-mod-1ae2b31a [e2e-llm-inference-service] - name: model-fb-opt-125m-with-lora-hf-c0d503b0 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:10:33Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:10:33Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:09:59Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:10:33Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:10:33Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://ab44bd37c915444139c8452d107668c4-1521421885.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-66746bd8b9-f297r to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:16:11 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:16:11 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:16:11 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.33:8000/health": dial tcp 10.132.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: autoscale-hpa-deploy-kserve-66746bd8b9-f297r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set autoscale-hpa-deploy-kserve-66746bd8b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/autoscale-hpa-deploy-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:12:57 kserve-ci-e2e-test LLMInferenceServiceController Warning ScalingCRDNotFound Required scaling CRD not installed: failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:54 kserve-ci-e2e-test LLMInferenceServiceController Warning Error Reconciliation failed: failed to reconcile workload: failed to reconcile scaling: failed to reconcile main workload scaling: failed to reconcile main VA: failed to get v1alpha1.VariantAutoscaling kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-va: no matches for kind "VariantAutoscaling" in version "llmd.ai/v1alpha1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/autoscale-hpa-deploy-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-concurrency-test-kserve-b5d44467c-xdjqc to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-concurrency-test-kserve-b5d44467c-xdjqc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:46 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-concurrency-test-kserve-router-scheduler-78c47c8645 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-concurrency-test-kserve-b5d44467c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/fc-concurrency-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-concurrency-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-concurrency-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/fc-concurrency-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-fc-concurrency-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/fc-concurrency-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/fc-concurrency-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-concurrency-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-concurrency-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/fc-concurrency-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:45:18 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-concurrency-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/fc-concurrency-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/fc-concurrency-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:45:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [fc-concurrency-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:36 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-concurrency-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:28 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test, llmisvc-model-fb-opt-125m-with-7ca60146, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-concurrency-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:16:13 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test, llmisvc-model-fb-opt-125m-with-ba4d693a, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-concurrency-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-smoke-test-kserve-7f4687897-r5tjx to ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" in 1.6s (1.6s including waiting). Image size: 79178094 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-smoke-test-kserve-7f4687897-r5tjx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.25/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:02 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" in 1.538s (1.538s including waiting). Image size: 179512283 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:03 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-smoke-test-kserve-router-scheduler-fcbfbd8bf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set fc-smoke-test-kserve-7f4687897 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/fc-smoke-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-smoke-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-smoke-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/fc-smoke-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-fc-smoke-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/fc-smoke-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/fc-smoke-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/fc-smoke-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/fc-smoke-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/fc-smoke-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:36 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with scheduler-v06-pd-migration-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/fc-smoke-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/fc-smoke-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [fc-smoke-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:57 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with scheduler-v06-threshold-migration-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with precise-prefix-cache-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:13 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with llmisvc-router-managed-test-llm-e45d1f79 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:15 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:50 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:35 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:25 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, llmisvc-model-fb-opt-125m-with-7ca60146, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:16:12 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, llmisvc-model-fb-opt-125m-with-ba4d693a, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/fc-smoke-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xj8kf to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:45 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:45 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:46 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.31:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-xj8kf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:32 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "gw-section-name-router-with-gat-2f0a622e": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "gw-section-name-router-with-gat-2f0a622e": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj4qcm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfj4qcm to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:06 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:06 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:06 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedMount MountVolume.SetUp failed for volume "tls-certs" : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74d68bf984-2dn6z to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:20 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:48 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 27.978s (27.978s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:48 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:48 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:08:56 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.59:8000/health": dial tcp 10.133.0.59:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:32 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:47 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.59:8000/health": dial tcp 10.133.0.59:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74d68bf984-2dn6z [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schem8sfx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-6c6dcbfc95 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schem8sfx to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:17 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:17 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:17 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:32 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:39 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74d68bf984 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:14 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:07 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:59 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:59 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:59 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:11:06 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.60:8000/health": dial tcp 10.133.0.60:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-7885885764 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:54 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:10:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:16:14 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-router-managed-test-llm-4b931143, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:10:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:10:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:11:16 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs to ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:10 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:36 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:47 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:28 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-model-fb-opt-125m-with-7ca60146, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:16:13 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-model-fb-opt-125m-with-ba4d693a, llmisvc-router-managed-test-llm-5b1e8f15 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2 to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:08 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:45 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:45 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:44:49 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:09:37 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-router-managed-test-llm-4b931143 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:07:23 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-model-fb-opt-125m-with-7ca60146, llmisvc-router-managed-test-llm-4b931143 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:16:14 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-concurrency-test, fc-smoke-test, llmisvc-model-fb-opt-125m-with-ba4d693a, llmisvc-router-managed-test-llm-4b931143 in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-sskh8 to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:44 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:44 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-sskh8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:05 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:43 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:43 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexb8dr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5657fd78d9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schexb8dr to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:10 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:44 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:34:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-6f9bc95cfd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:14 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:14 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:51:14 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:35:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-69b67488bb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:08 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/model-fb-opt-125m-gw-section-na-8bf35153: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:35 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/model-fb-opt-125m-gw-section-na-e3d39686: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d6c8b5f7c-kgjrz to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" in 4.327s (4.327s including waiting). Image size: 300595129 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:02 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:02 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:02 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d6c8b5f7c-nmsvz to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:57 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6d6c8b5f7c-nmsvz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6d6c8b5f7c-kgjrz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-75779d4bffhzf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75779d4bffhzf to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b" in 4.016s (4.016s including waiting). Image size: 300595129 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:58 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:04 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:05 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:05 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:59 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/llm-d-kv-cache:v0.8.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:43 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/llm-d-kv-cache:v0.8.0" in 43.867s (43.867s including waiting). Image size: 2935159967 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:43 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:43 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.54:8082/healthz": dial tcp 10.133.0.54:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:04 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning BackOff Back-off restarting failed container main in pod precise-prefix-cache-test-kserve-router-scheduler-75779d4bffhzf_kserve-ci-e2e-test(9ff96acc-75a8-48e2-9709-8abb012acf3f) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:15 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.133.0.54:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:40 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: Get "http://10.133.0.54:8082/healthz": dial tcp 10.133.0.54:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:48 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-75779d4b5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-6d6c8b5f7c from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:39 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/precise-prefix-cache-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:32:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:38 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:33:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-kfbtn to ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:10 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:10 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:10 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:16 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.32:15021/healthz/ready": dial tcp 10.134.0.32:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-qbmpr to ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-qbmpr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-qbmpr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-kfbtn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gx98l to ip-10-0-133-160.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:41 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:41 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:41 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:39 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Warning Unhealthy Readiness probe failed: Get "http://10.134.0.30:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:46 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.30:15021/healthz/ready": dial tcp 10.134.0.30:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test kubelet/ip-10-0-133-160.ec2.internal Warning Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-gx98l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-gx98l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:21 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:24 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-54c47cdfc7 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-799f46c59b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:34 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:07 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/router-with-gateway-ref-gw-sect-bd0c4d94: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:33 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/router-with-gateway-section-nam-5d237677: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:34 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/router-with-managed-route-gw-se-010f4010: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:08 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/router-with-managed-route-gw-se-bc1f68e4: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-68f9854ff-82kzx to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.24/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" in 2.361s (2.361s including waiting). Image size: 166803393 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:51 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" in 1.483s (1.483s including waiting). Image size: 79178094 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:52 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-68f9854ff-82kzx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-4cx7q to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:49 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" in 1.754s (1.754s including waiting). Image size: 79178094 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:49 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:49 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.49:8000/health": dial tcp 10.133.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-4cx7q [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-router-scheduler-79ww96l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-79ww96l to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:48 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" in 2.55s (2.55s including waiting). Image size: 179512283 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:50 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:42 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-router-scheduler-79c5847dfd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-68f9854ff from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-pd-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-pd-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:28:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:29 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/scheduler-v06-pd-migration-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:29:28 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-v06-pd-migration-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5d54f65f57-27ctz to ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.26/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-128-42.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-5d54f65f57-27ctz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-prefill-b58fmzjxs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-b58fmzjxs to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-prefill-b58fbd55d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6ws58 to ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning FailedToRetrieveImagePullSecret Unable to retrieve some image pull secrets (scheduler-v06-threshold-migration-test-epp-sa-dockercfg-klsww); attempting to pull the image may not succeed. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:38 kserve-ci-e2e-test kubelet/ip-10-0-132-130.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-router-sched6ws58 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-router-scheduler-5db9bd4595 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-5d54f65f57 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-threshold-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-threshold-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:29 kserve-ci-e2e-test LLMInferenceServiceController Warning ModelNameCollision model name "facebook/opt-125m" overlaps with fc-smoke-test in namespace kserve-ci-e2e-test; shared publisher paths and model-routing headers cause the gateway to shadow one service. Reachable at its own address kserve-ci-e2e-test/scheduler-v06-threshold-migration-test; safe to ignore if traffic splitting is planned [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:29 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-v06-threshold-migration-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 17:31:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:00:35 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/workload-llmd-simulator-gw-sect-0fb35c52: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-16 18:01:09 kserve-ci-e2e-test LLMInferenceServiceConfigController Warning DeletionBlocked Cannot delete LLMInferenceServiceConfig kserve-ci-e2e-test/workload-llmd-simulator-gw-sect-81946990: referenced by 1 LLMInferenceService(s) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 2026-07-16 18:09:56.603 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models'), ('hf://edbeeching/opt-125m-lora', '/mnt/lora/lora-adapter-1')] [e2e-llm-inference-service] 2026-07-16 18:09:56.603 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] 2026-07-16 18:09:56.622 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():449] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmpncv4ugdw.crt [e2e-llm-inference-service] 2026-07-16 18:09:59.356 1 storage.initializer INFO [kserve_storage.py:download():243] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-07-16 18:09:59.356 1 storage.initializer INFO [kserve_storage.py:download():244] Model downloaded in 2.752306451000095 seconds. [e2e-llm-inference-service] 2026-07-16 18:09:59.356 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of hf://edbeeching/opt-125m-lora to local [e2e-llm-inference-service] 2026-07-16 18:09:59.714 1 storage.initializer INFO [kserve_storage.py:download():243] Successfully copied hf://edbeeching/opt-125m-lora to /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] 2026-07-16 18:09:59.714 1 storage.initializer INFO [kserve_storage.py:download():244] Model downloaded in 0.3583037180001156 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 (APIServer pid=1) DEBUG 07-16 18:23:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:33 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:34 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:34 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:37 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:38 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:40 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:41 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:43 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:44 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:44 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:47 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:48 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:50 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:51 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:53 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:54 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:54 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:57 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:58 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:23:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:00 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:01 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:03 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:04 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:04 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:07 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:08 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:10 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:11 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:13 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:14 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:14 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:17 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:18 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:20 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:21 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:23 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:24 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:24 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:27 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:28 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:30 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:31 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:33 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:34 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:34 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:37 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:38 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:40 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:41 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:43 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:44 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:44 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:47 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:48 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:50 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:51 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:53 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:54 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:54 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:57 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:58 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:24:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:00 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:01 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:03 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:04 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:04 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:07 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:08 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:10 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:11 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:13 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:14 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:14 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:17 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:18 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:19 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:20 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:21 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:22 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:23 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:24 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:24 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:25 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:26 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:27 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:28 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:29 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:30 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:31 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:32 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:33 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:34 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:34 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:35 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:36 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:37 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:38 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:39 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:40 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:41 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:42 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:43 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:44 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:44 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:45 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:46 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:47 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:48 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:49 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:50 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:51 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:52 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:53 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:54 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:54 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:55 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:56 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:57 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:58 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:25:59 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:00 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:01 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:02 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:03 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:04 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:04 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:07 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:08 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:10 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:11 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:13 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:14 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:14 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-16 18:26:17 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 2026-07-16 18:09:56.916 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-07-16 18:09:56.917 1 storage.initializer INFO [kserve_storage.py:download():168] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] 2026-07-16 18:09:56.917 1 storage.initializer INFO [kserve_storage.py:download():171] Allow patterns: ['tokenizer.json', 'tokenizer_config.json', 'special_tokens_map.json', 'vocab.json', 'merges.txt', 'config.json', 'generation_config.json'] [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_5e1d30fe-7040-4236-aaaa-e88b16c03049'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_d6a007f6-9866-49ab-9e21-c76e9ed8f292'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_f2ffe5f0-22e7-4960-89b8-ecc729d7ed1b'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_855cda15-f2a0-4236-b691-87800e540ff7'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_8e139ab8-391f-495f-bed4-678fabcd7d65'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_a580ccdb-0483-41ca-95b6-b8a8ec414b0f'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-07-16 18:09:57.327 1 storage.initializer INFO [kserve_storage.py:download():243] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-07-16 18:09:57.327 1 storage.initializer INFO [kserve_storage.py:download():244] Model downloaded in 0.4107198290002998 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1784225397.700063,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1784225397.7002933,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n- type: lora-affinity-scorer\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: lora-affinity-scorer\n weight: 4\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n weight: 2\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7006216,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7015703,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7016194,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: prefix-cache-scorer} {Type: no-hit-lru-scorer} {Type: max-score-picker} {Type: lora-affinity-scorer}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: lora-affinity-scorer, Weight: 4.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: no-hit-lru-scorer, Weight: 2.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.701631,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7018833,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"no-hit-lru-scorer","parameters":null,"type":"no-hit-lru-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"lora-affinity-scorer","parameters":null,"type":"lora-affinity-scorer"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"lora-affinity-scorer","weight":4},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"no-hit-lru-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1784225397.720142,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1784225397.72023,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.72027,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7203069,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7203658,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7203856,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7204428,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: no-hit-lru-scorer, Type: no-hit-lru-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: lora-affinity-scorer, Type: lora-affinity-scorer} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: lora-affinity-scorer, Weight: 4.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: no-hit-lru-scorer, Weight: 2.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7205043,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7205648,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.720588,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"no-hit-lru-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7206087,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7207036,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7207134,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [lora-affinity-scorer/lora-affinity-scorer: 4.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc0003e4fc0 DataConfig:{Sources:[{Plugin:0xc0004baf30 Extractors:[0xc0003e51c0]}]} FlowControlConfig: ParserRegistry:0xc0003e5740}"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7400281,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.740056,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.740073,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7400799,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7400858,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7400925,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7400982,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7401223,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7401292,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.740138,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7401454,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.740153,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7401648,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7401736,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [lora-affinity-scorer/lora-affinity-scorer: 4.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.740212,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7402244,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7402477,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1784225397.740267,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7402816,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.740365,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7403827,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7404273,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7407665,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7407713,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7408526,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1784225397.741045,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7412002,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7412114,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.741631,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7419,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7419763,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7465324,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7465336,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7468746,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.7489216,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.8416696,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.8416893,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784225397.8416936,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.84171,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784225397.9418514,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.9418817,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784225397.942009,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","reconcileID":"23ef2b41-e588-4351-a4df-ab774c905349"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.9429033,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1784225397.9429235,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1784225427.75755,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","reconcileID":"e046e48d-8fc8-44ea-b78b-5f0e38d5cc0b"} [e2e-llm-inference-service] {"level":"info","ts":1784225476.2240112,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r","reconcileID":"d3ffb5ac-c27b-4730-a876-7ea4a0c2fbc7"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 874fae1e-9b48-496a-a6be-cb9fc059d94b [e2e-llm-inference-service] resourceVersion: '51628' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.35 [e2e-llm-inference-service] nodeName: ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g [e2e-llm-inference-service] uid: 98d3963f-09e7-4690-aa9e-b998e764b139 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 04f21642-601f-426c-b5c4-4b0b375b0771 [e2e-llm-inference-service] resourceVersion: '52182' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.60 [e2e-llm-inference-service] nodeName: ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r [e2e-llm-inference-service] uid: d8e3525f-b61f-4ed2-ad06-a136527b145c [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d8e3525f-b61f-4ed2-ad06-a136527b145c [e2e-llm-inference-service] resourceVersion: '52181' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 9bddccd4c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.60/23"],"mac_address":"0a:58:0a:85:00:3c","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.60/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.60\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:3c\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c [e2e-llm-inference-service] uid: f7cb8714-66e4-4543-9450-a781482cfc8c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-132-130 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f7cb8714-66e4-4543-9450-a781482cfc8c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.60"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-tsqt9 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] - name: kube-api-access-tsqt9 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: kube-api-access-tsqt9 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:09:59Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] hostIP: 10.0.132.130 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.132.130 [e2e-llm-inference-service] podIP: 10.133.0.60 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.60 [e2e-llm-inference-service] startTime: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] finishedAt: '2026-07-16T18:09:59Z' [e2e-llm-inference-service] containerID: cri-o://dcf5cf29e5693d6c37126b5ca3260a1f89d6d10cc42e33d4e047409c58500be2 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] containerID: cri-o://dcf5cf29e5693d6c37126b5ca3260a1f89d6d10cc42e33d4e047409c58500be2 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-tsqt9 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-16T18:09:59Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0 [e2e-llm-inference-service] containerID: cri-o://f888d24b6db9e475c2466382df41b81cf2ea94da280b979d8b57f147b0605e4b [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-tsqt9 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-7885885764- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 98d3963f-09e7-4690-aa9e-b998e764b139 [e2e-llm-inference-service] resourceVersion: '51627' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: '7885885764' [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.35/23"],"mac_address":"0a:58:0a:84:00:23","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.35/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.35\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:23\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-7885885764 [e2e-llm-inference-service] uid: 040d4f40-995c-492f-b4ee-044a6103d45d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-42 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"040d4f40-995c-492f-b4ee-044a6103d45d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.35"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-8kvw2 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-8kvw2 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - type: lora-affinity-scorer\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: lora-affinity-scorer\n weight: 4\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-8kvw2 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-d49nw [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] hostIP: 10.0.128.42 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.42 [e2e-llm-inference-service] podIP: 10.132.0.35 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.35 [e2e-llm-inference-service] startTime: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] finishedAt: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] containerID: cri-o://8746d83f43ceb4c04e599f741b5d851349a09de8b5be0a5a8ecfcf6fd1a8dfbb [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] containerID: cri-o://8746d83f43ceb4c04e599f741b5d851349a09de8b5be0a5a8ecfcf6fd1a8dfbb [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-8kvw2 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://6cd9a2c397f091f09d64f1b9cdb626e2549ed3325f76508b17b6f90750b90a9a [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-8kvw2 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a278411d-7425-4ecd-a074-27aba40168fa [e2e-llm-inference-service] resourceVersion: '51049' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-d49nw [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-d49nw"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-sdm8r"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-d49nw [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-sdm8r [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-d49nw [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 29132334-b317-4ef2-8305-6ca7bee97cd9 [e2e-llm-inference-service] resourceVersion: '51069' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.3.176 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.3.176 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ab76431d-6ccf-4dab-885d-432f24ad5818 [e2e-llm-inference-service] resourceVersion: '51041' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.123.116 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.123.116 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d230dea9-e17e-4df0-b8f2-f67c15f11150 [e2e-llm-inference-service] resourceVersion: '52187' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f54d96b5-6793-4b04-bc25-06b071044d62 [e2e-llm-inference-service] resourceVersion: '51631' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n- type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\n- type: lora-affinity-scorer\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: lora-affinity-scorer\n weight:\ [e2e-llm-inference-service] \ 4\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ no-hit-lru-scorer\n weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-7885885764" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f7cb8714-66e4-4543-9450-a781482cfc8c [e2e-llm-inference-service] resourceVersion: '52186' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 9bddccd4c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] uid: d230dea9-e17e-4df0-b8f2-f67c15f11150 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d230dea9-e17e-4df0-b8f2-f67c15f11150"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 9bddccd4c [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 9bddccd4c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-7885885764 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 040d4f40-995c-492f-b4ee-044a6103d45d [e2e-llm-inference-service] resourceVersion: '51630' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: '7885885764' [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] uid: f54d96b5-6793-4b04-bc25-06b071044d62 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f54d96b5-6793-4b04-bc25-06b071044d62"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: '7885885764' [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: '7885885764' [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:735ced4d2d869072184c156c722fd9f0956b6fb71ca85783529cf832d9acbf7b [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n- type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\n- type: lora-affinity-scorer\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: lora-affinity-scorer\n weight:\ [e2e-llm-inference-service] \ 4\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ no-hit-lru-scorer\n weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cfe374ec-23de-4966-b509-65de0339db18 [e2e-llm-inference-service] resourceVersion: '51060' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 19d1e238-0777-45e0-8912-93861e757272 [e2e-llm-inference-service] resourceVersion: '51057' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service-t2gp5 [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 442b20cb-b058-4795-b687-adfc375b33d4 [e2e-llm-inference-service] resourceVersion: '51629' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] uid: 29132334-b317-4ef2-8305-6ca7bee97cd9 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"29132334-b317-4ef2-8305-6ca7bee97cd9"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.35 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g [e2e-llm-inference-service] uid: 98d3963f-09e7-4690-aa9e-b998e764b139 [e2e-llm-inference-service] nodeName: ip-10-0-128-42.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-sv92nc2 [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 52c81bea-102c-4a15-9337-c036b9fbfff3 [e2e-llm-inference-service] resourceVersion: '52185' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] uid: ab76431d-6ccf-4dab-885d-432f24ad5818 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T18:11:16Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ab76431d-6ccf-4dab-885d-432f24ad5818"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.60 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r [e2e-llm-inference-service] uid: d8e3525f-b61f-4ed2-ad06-a136527b145c [e2e-llm-inference-service] nodeName: ip-10-0-132-130.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cfe374ec-23de-4966-b509-65de0339db18 [e2e-llm-inference-service] resourceVersion: '51060' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 19d1e238-0777-45e0-8912-93861e757272 [e2e-llm-inference-service] resourceVersion: '51057' [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-16T18:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:10:27Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T18:10:27Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51634' [e2e-llm-inference-service] uid: dfcda096-d7ed-44a6-9b0e-8b14d69728a2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:10:27Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:09:58Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:10:27Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T18:10:27Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T18:10:29Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51634' [e2e-llm-inference-service] uid: dfcda096-d7ed-44a6-9b0e-8b14d69728a2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:10:27Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:09:58Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T18:10:27Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51596' [e2e-llm-inference-service] uid: d529faa3-c2fd-4a29-98a0-44b31736183e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:10:27Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:10:27Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-16T18:10:02Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51292' [e2e-llm-inference-service] uid: 3daaab11-f32d-4845-af63-63d50d831afa [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:10:00Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-16T18:10:02Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51110' [e2e-llm-inference-service] uid: ab03a645-8eb0-4908-9e4f-0b96c48311b9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:10:28Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:10:28Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51608' [e2e-llm-inference-service] uid: 844b54b1-a321-4417-8786-461f71c1d376 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51123' [e2e-llm-inference-service] uid: ffc9cebb-e916-4892-a88d-a25ab775b49a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51110' [e2e-llm-inference-service] uid: ab03a645-8eb0-4908-9e4f-0b96c48311b9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:10:28Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:10:28Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51608' [e2e-llm-inference-service] uid: 844b54b1-a321-4417-8786-461f71c1d376 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51123' [e2e-llm-inference-service] uid: ffc9cebb-e916-4892-a88d-a25ab775b49a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51110' [e2e-llm-inference-service] uid: ab03a645-8eb0-4908-9e4f-0b96c48311b9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:10:28Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:10:28Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51608' [e2e-llm-inference-service] uid: 844b54b1-a321-4417-8786-461f71c1d376 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:09:57Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51123' [e2e-llm-inference-service] uid: ffc9cebb-e916-4892-a88d-a25ab775b49a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5fbd197-5d06-4509-acec-6a89d4ec1cbe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-16T18:09:56Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: d5fbd197-5d06-4509-acec-6a89d4ec1cbe [e2e-llm-inference-service] resourceVersion: '51083' [e2e-llm-inference-service] uid: 82865c1f-6542-49bf-8bbf-bde6e3aa4db4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:26:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 9bddccd4c [e2e-llm-inference-service] timestamp: '2026-07-16T18:25:55Z' [e2e-llm-inference-service] window: 15.588s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 100535796n [e2e-llm-inference-service] memory: 2319564Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-16T18:26:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: '7885885764' [e2e-llm-inference-service] timestamp: '2026-07-16T18:26:02Z' [e2e-llm-inference-service] window: 18.362s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 55038830n [e2e-llm-inference-service] memory: 33464Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-16T18:26:19.356539] end - ❌ 987.050s: Service returned 401: [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:240: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:240: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:240: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:240: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:240: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:240: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:240: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:240: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] ---------- generated xml file: /workspace/artifacts-dir/junit_e2e.xml ---------- [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] FAILED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] ERROR llmisvc/test_llm_auth.py [e2e-llm-inference-service] ERROR llmisvc/test_llm_tls.py [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 8 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] == 6 failed, 7 passed, 3 skipped, 12 warnings, 2 errors in 3470.12s (0:57:50) == [must-gather] [must-gather ] OUT 2026-07-16T18:26:24.510856803Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: a9b59be7-cab2-4d29-b5e3-23627224b3ec [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-16T18:26:24.551219371Z namespace/openshift-must-gather-j9rsm created [must-gather] [must-gather ] OUT 2026-07-16T18:26:24.556373407Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-542ng created [must-gather] [must-gather ] OUT 2026-07-16T18:26:24.578746964Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:31.826396321Z [disk usage checker] Started [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:31.829834116Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.140079549Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.301092128Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.301132749Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.301132749Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.303549966Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.463386846Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.463419257Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.463419257Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.465981002Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.626541631Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.626665869Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.626665869Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.629102900Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.789632665Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.789663367Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.789663367Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.791835202Z Error getting logs from rhods-notebooks [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.966868429Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.966908916Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.966908916Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:32.970261835Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.155612749Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.155644230Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.155644230Z namespaces "istio-system" not found [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.157802184Z Error getting logs from istio-system [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.315498127Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.315616459Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.315616459Z namespaces "knative-serving" not found [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.317706074Z Error getting logs from knative-serving [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.473975807Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.474004243Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.474004243Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.476073655Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:33.746735302Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.054176593Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.215462984Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.312521168Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.314483426Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.442199656Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.524238402Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.526135145Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.656444424Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.741857831Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.744034307Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.873448237Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.958859649Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:34.960896523Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.108256444Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.216494578Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.218728329Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.378117681Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.462528898Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.464686068Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.594668976Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.677385245Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.679458568Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.809037729Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.895526177Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:35.897893659Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.028516702Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.112496120Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.115683923Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.250964677Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.336259775Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.338500927Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.486974266Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.580506348Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.582741480Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.712407551Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.796089285Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.798406171Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.834755151Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:36.929985688Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.014175780Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.016281536Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.153598787Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.265178798Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.267387654Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.410300853Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.500110713Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.502343002Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.670657360Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.758518424Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.760945626Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.891694121Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.974255845Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:37.976567030Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:38.112560251Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:38.194025115Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:38.196134017Z Error collecting info from [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:38.942264490Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:39.100362711Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:39.239456510Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:39.368527396Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:39.502905461Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:40.147029921Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:40.277766413Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:40.409012000Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:40.542866735Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:40.676711808Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:41.302751150Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:41.842162424Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:41.997160250Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:42.388819125Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:42.584567083Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:42.747099132Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:43.408535940Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:43.792840021Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:43.994770544Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:44.199428943Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:44.718974640Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:45.093319068Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:45.326677258Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:45.524028228Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:46.041920773Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:46.416059314Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:46.651197351Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:46.821741704Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:46.847279613Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:47.393143621Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:47.864274653Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:48.078136223Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:48.237705000Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:48.826006312Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:49.239985859Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:49.448628002Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:49.617250452Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:50.155747864Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:50.527501783Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:50.732441097Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:50.900215177Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:51.445949106Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:51.817046701Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:51.852065338Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:52.042267018Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:52.244479923Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:52.778653108Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:53.170169757Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:53.397251258Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:53.565062669Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:54.071977915Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:54.451886405Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:54.679618562Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:54.839759413Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:55.355701874Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:55.781118963Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:55.978049293Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:56.130722078Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:56.653058255Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:56.857494813Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:57.055162343Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:57.247698772Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:57.407726368Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:57.956966038Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:58.363094466Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:58.559353587Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:58.721446893Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:59.243526379Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:59.633074997Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:59.820907120Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:26:59.976387901Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:00.523616838Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:00.891045650Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:01.082157441Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:01.242549916Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:01.864253011Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:01.938911098Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:02.092937395Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:02.279736054Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:02.437744606Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:03.151128402Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:03.316607954Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:03.510622312Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:03.664050824Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:04.367280829Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:04.506388509Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:04.695499761Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:04.848886968Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:05.559850714Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:05.703576356Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:05.895538624Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:06.049484143Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:06.497379884Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:06.693168636Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:06.871352429Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:06.946503209Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:07.103532228Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:07.567954136Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:07.779490129Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:08.011945364Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:08.182618518Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:08.619228391Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:08.798324079Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:09.018707493Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:09.213665362Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:09.661099681Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:09.837401454Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:10.041068382Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:10.205056100Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:10.660046577Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:10.838010509Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:11.039695867Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:11.193532610Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:11.655362358Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:11.834435514Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:11.877085325Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:12.030752655Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:12.181730980Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:12.625751596Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:12.822575541Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:13.048570670Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:13.201897702Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:13.646318185Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:13.834763792Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:14.113949578Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:14.271121065Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:14.700425684Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:14.885078129Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:15.090200106Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:15.264181234Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:15.702362892Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:15.888215617Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:16.114063682Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:16.303352518Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:16.761566244Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:16.881610891Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:16.947901454Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:17.164995764Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:17.328238863Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:17.789891379Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:17.968706841Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:18.179217866Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:18.350138456Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:18.805783028Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:18.984871143Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:19.186346848Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:19.352819358Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:19.818179101Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:20.004951714Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:20.218313157Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:20.366149913Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:20.888213261Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:21.064638252Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:21.268973418Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:21.422442325Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:21.854571737Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:21.886019147Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:22.043974984Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:22.261989119Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:22.413280619Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:22.856306979Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:23.042669725Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:23.269716741Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:23.442160129Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:23.890643865Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:24.070095453Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:24.282515586Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:24.456688687Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:24.926089107Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:25.102875767Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:25.305432856Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:25.461804755Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:25.937769880Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:26.115782099Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:26.324452555Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:26.480283086Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:26.891848674Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:26.952483090Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:27.140088171Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:27.343348383Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:27.495456240Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:27.940714433Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:28.129244803Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:28.379829294Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:28.529564921Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:28.966652663Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:29.168974520Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:29.394693386Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:29.560088786Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:29.992987457Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:30.183810186Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:30.426456903Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:30.605611118Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:31.058582142Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:31.265954130Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:31.492533110Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:31.692063466Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:31.896368779Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:32.129008889Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:32.305665171Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:32.510321394Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:32.668212620Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:33.137058047Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:33.311925920Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:33.515141508Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:33.674509263Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:34.189423657Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:34.366671285Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:34.565535885Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:34.743011416Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:35.218475709Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:35.398936969Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:35.604353055Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:35.769992650Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:36.231175468Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:36.404340436Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:36.605425040Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:36.758248106Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:36.900631214Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:37.234122017Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:37.499476912Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:37.713067066Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:37.862973442Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:38.359833118Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:38.581169328Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:38.793682594Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:38.954717876Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:39.443127459Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:39.695920554Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:39.913412062Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:40.063977253Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:40.493194855Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:40.674466533Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:40.910849191Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:41.088874517Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:41.543001528Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:41.729901231Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:41.905536049Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:41.956459869Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:42.130124887Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:42.584758539Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:42.763944538Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:42.979474280Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:43.153024714Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:43.621930139Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:43.803089647Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:44.010608008Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:44.167935747Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:44.636466086Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:44.809011905Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:45.016725588Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:45.171630778Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:45.649656660Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:45.824515492Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:46.031245757Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:46.182196574Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:46.647423750Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:46.873015481Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:46.910741485Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:47.073311033Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:47.227440072Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:47.695466984Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:47.925900524Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:48.132602543Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:48.288398615Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:48.725395274Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:48.925043726Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:49.136015417Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:49.299461960Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:49.733551850Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:49.941024727Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:50.150740258Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:50.310734477Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:50.759223873Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:50.932897574Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:51.138045695Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:51.286749405Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:51.762242294Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:51.916466647Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:51.937563255Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:52.134015986Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:52.283200045Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:52.747751357Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:53.009071454Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:53.212880542Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:53.363327874Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:53.820193619Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:54.033923004Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:54.259952085Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:54.412767350Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:54.859492703Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:55.072514834Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:55.302620411Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:55.454925920Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:55.890624795Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:56.091382261Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:56.346429535Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:56.514668483Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:56.922842305Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:56.976937395Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:57.154438920Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:57.372060532Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:57.572596803Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:58.022077904Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:58.208117931Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:58.430562215Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:58.605369387Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:59.092730971Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:59.279303263Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:59.491410521Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:27:59.653597457Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:00.098579771Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:00.275110000Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:00.480524798Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:00.643210054Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:01.131337094Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:01.312764591Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:01.522664804Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:01.687691035Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:01.927312500Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:02.170819788Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:02.369813641Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:02.584160395Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:02.745397564Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:03.206101067Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:03.400760272Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:03.622441275Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:03.787016180Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:04.230677866Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:04.421264198Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:04.678017636Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:04.839051032Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:05.279121860Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:05.479965903Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:05.724733791Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:05.900133170Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:06.368765526Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:06.545445201Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:06.774331457Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:06.932656771Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:06.992202154Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:07.474990691Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:07.673946389Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:07.887341497Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:08.049318842Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:08.493925203Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:08.683934540Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:08.895669618Z Wrote inspect data to must-gather. [must-gather] [must-gather-w7m6q] POD 2026-07-16T18:28:08.917987818Z Caches written to disk [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:09.926370742Z waiting for gather to complete [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:09.929740132Z downloading gather output [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.214842959Z receiving incremental file list [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.231880633Z ./ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.231964328Z aggregated-discovery-api.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.232275149Z aggregated-discovery-apis.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.233248612Z event-filter.html [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.234373806Z timestamp [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.234577849Z version [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.239983733Z cluster-scoped-resources/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.239998734Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.240003704Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.24009539Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.240189176Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.240197377Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.24025187Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.240353067Z namespaces/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.240361697Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.240447693Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.240530428Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.240614304Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.240671968Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.240724691Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.240832818Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.241489941Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.242302735Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.242363968Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.242449554Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.242522339Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.242580222Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.242685629Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.242746873Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.242800527Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.242895433Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.242954997Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.243011721Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.243171251Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.243369174Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.245645853Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.24575227Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.246494178Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.246661179Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.247065886Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.247276029Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.247331223Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.247550518Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.247601411Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.247666115Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.247701607Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.247812435Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.247874488Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.247940193Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248020408Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248071981Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248132796Z namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248139676Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-b5d44467c-xdjqc/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248185259Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-b5d44467c-xdjqc/fc-concurrency-test-kserve-b5d44467c-xdjqc.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248310657Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-b5d44467c-xdjqc/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248320468Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-b5d44467c-xdjqc/main/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248326618Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-b5d44467c-xdjqc/main/main/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248372171Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-b5d44467c-xdjqc/main/main/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248548272Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-b5d44467c-xdjqc/main/main/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248652199Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-b5d44467c-xdjqc/main/main/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248677501Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248749426Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248873944Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248904156Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx/main/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248912086Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx/main/main/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.248922497Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx/main/main/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.249113689Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx/main/main/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.249206576Z namespaces/kserve-ci-e2e-test/pods/fc-concurrency-test-kserve-router-scheduler-78c47c8645-m62qx/main/main/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.249255959Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-7f4687897-r5tjx/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.249288311Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-7f4687897-r5tjx/fc-smoke-test-kserve-7f4687897-r5tjx.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.24942808Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-7f4687897-r5tjx/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.249447401Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-7f4687897-r5tjx/main/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.249453142Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-7f4687897-r5tjx/main/main/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.249479033Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-7f4687897-r5tjx/main/main/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.249618802Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-7f4687897-r5tjx/main/main/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.249714069Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-7f4687897-r5tjx/main/main/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.249748441Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.24988368Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.249992037Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.250000818Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk/main/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.250005368Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk/main/main/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.250054981Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk/main/main/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25201773Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk/main/main/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.252087614Z namespaces/kserve-ci-e2e-test/pods/fc-smoke-test-kserve-router-scheduler-fcbfbd8bf-k8qnk/main/main/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.252105525Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.252163669Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25232675Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.252339871Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/main/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.252345541Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/main/main/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.252352961Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/main/main/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25294074Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/main/main/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253007364Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/main/main/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253028745Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/storage-initializer/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253034066Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/storage-initializer/storage-initializer/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253041966Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253070158Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253183266Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.2532525Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-9bddccd4c-z5w8r/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253271311Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253320675Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253481535Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253496066Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/main/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253500726Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/main/main/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253538679Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/main/main/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253680828Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/main/main/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253739892Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/main/main/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253781805Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/storage-initializer/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253787995Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/storage-initializer/storage-initializer/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253827758Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253835498Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253919154Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.253986098Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schewz48g/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254028741Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254056613Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254156039Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25416346Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs/main/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25416746Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs/main/main/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254210403Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs/main/main/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254354382Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs/main/main/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254493051Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-4b931143-kserve-85bf86796966gbs/main/main/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254551365Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254603468Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254706415Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254714186Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2/main/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254718406Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2/main/main/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25477505Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2/main/main/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254904008Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2/main/main/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.254995634Z namespaces/kserve-ci-e2e-test/pods/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-xdvj2/main/main/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255049738Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255149354Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255254211Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255269352Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf/main/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255274242Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf/main/main/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255321215Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf/main/main/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255520248Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf/main/main/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255616055Z namespaces/kserve-ci-e2e-test/pods/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep28gf/main/main/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255668898Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255716781Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255827379Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255836589Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk/main/main/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25584314Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk/main/main/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.255889023Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk/main/main/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256057443Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk/main/main/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25614788Z namespaces/kserve-ci-e2e-test/pods/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche7q2hk/main/main/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256184512Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-kfbtn/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256253726Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-kfbtn/router-gateway-1-openshift-default-54c47cdfc7-kfbtn.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256386655Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-kfbtn/istio-proxy/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256416747Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-kfbtn/istio-proxy/istio-proxy/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256427768Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-kfbtn/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25647079Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-kfbtn/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256630111Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-kfbtn/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256727167Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-kfbtn/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256779941Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256828014Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256889678Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.256940241Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.257000005Z namespaces/kuadrant-system/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.257059149Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.257136834Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.257186917Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.257254792Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.257305175Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.257424713Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.257891443Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.258206434Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.258265548Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.258322011Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.258381195Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.258490762Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.258616291Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.258692196Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.258744169Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.258848916Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25890628Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.258962183Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25922122Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25936756Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.259874873Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.25998916Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.260349794Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.260525776Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.260800744Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.260944983Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.260986326Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261141656Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261188109Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261251063Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261309417Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261419544Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261496879Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261549843Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261739445Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261776127Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261848742Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261861683Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.261906216Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino-7c77957575-xc96s.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.262000722Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.262008633Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/authorino/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.262013393Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/authorino/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.262056886Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/authorino/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.265519582Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.265612648Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/authorino/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.265654921Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.265718745Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/authorino-operator-6d85f6564-c4d5z.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.265832673Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.265840063Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/manager/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.265844133Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/manager/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.265890406Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/manager/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266106431Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266197666Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/manager/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.26624514Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266290403Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/dns-operator-controller-manager-65b49595d7-m7vkn.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266421771Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266434622Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/manager/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266440622Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/manager/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266479815Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/manager/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266677708Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266775324Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/manager/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266818987Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266884371Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin-85fd4b8b9d-m6wrp.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266974897Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266992798Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.266997739Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.267042362Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.267134258Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.267243925Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.267285897Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.267346952Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.267495331Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.267506952Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/manager/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.267511112Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/manager/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.267563156Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/manager/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.271885598Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.271980035Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/manager/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272030338Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272081241Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador-limitador-69574b596d-nzkh5.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.27221198Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.27222119Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/limitador/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272226091Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/limitador/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272290355Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/limitador/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272389871Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272508029Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/limitador/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272556112Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272614896Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/limitador-operator-controller-manager-6f9f468797-d7wdw.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272718203Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272726373Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/manager/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272739004Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/manager/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272783357Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/manager/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.272971819Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.273064975Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/manager/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.273113789Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.273156291Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.273223326Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.27328625Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.273345744Z namespaces/openshift-ingress/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.273427049Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.273530526Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.2735887Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.273646163Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.273708027Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.273815184Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.274093953Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.274653839Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.274697522Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.274759536Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.274852882Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.274877644Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.275006252Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.275047245Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.27511923Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.275199345Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.275260739Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.275308422Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.275869449Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.276002497Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.276250614Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.276474498Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.276775748Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.276931548Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.277180804Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.277287722Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.277339755Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.277463433Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.277508356Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.27757611Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.277617873Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.27772462Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.277775013Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.277841287Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.277916963Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.277962795Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.27802656Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.278038891Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.278078393Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/istiod-openshift-gateway-94bb8fbfd-qnh6j.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.278216322Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.278224952Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.278230193Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.278274776Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.288654174Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.28874472Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.288800414Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.288853457Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.288995577Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.289011558Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.289017218Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.289052951Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291218152Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291312028Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291368572Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router-default-568468f9f4-9h48n.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291534503Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291549954Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291562984Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291592797Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/current.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291734936Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/previous.insecure.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291819121Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/previous.log [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291873065Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291937699Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.291988212Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.292042146Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.301557018Z [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.30158786Z sent 4,014 bytes received 958,915 bytes 641,952.67 bytes/sec [must-gather] [must-gather-w7m6q] OUT 2026-07-16T18:28:10.30159456Z total size is 11,291,962 speedup is 11.73 [must-gather] [must-gather ] OUT 2026-07-16T18:28:10.428430262Z namespace/openshift-must-gather-j9rsm deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: a9b59be7-cab2-4d29-b5e3-23627224b3ec [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-16T18:28:10.580480982Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec435553f79197f28a94d55da851b8302383238997d790ed3335149e3ee4341f [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: a9b59be7-cab2-4d29-b5e3-23627224b3ec [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-16T18:28:10.600609637Z namespace/openshift-must-gather-qh77t created [must-gather] [must-gather ] OUT 2026-07-16T18:28:10.605340327Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-ptjhh created [must-gather] [must-gather ] OUT 2026-07-16T18:28:10.62665305Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec435553f79197f28a94d55da851b8302383238997d790ed3335149e3ee4341f created [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:13.088196789Z [disk usage checker] Started [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:13.091820235Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:13.311443214Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:13.485746840Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:13.598995213Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:13.706333681Z Gathering data for ns/default... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:14.090658947Z Gathering data for ns/openshift... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:14.608462259Z Gathering data for ns/kube-system... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:15.406861555Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:15.836324014Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:15.836418244Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:15.836418244Z namespaces "assisted-installer" not found [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.137395216Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.143147871Z INFO: Collecting host service logs for crio [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.143475156Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.143810180Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.144316750Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.144669514Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.144784008Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.145009308Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.145373691Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.145485079Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.145774769Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.146140148Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.146486389Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.146841103Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.147458430Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.151360129Z INFO: Gathering HAProxy config files [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.160799161Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.184270870Z Executing Istio gather script [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.285154764Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.485941863Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.919319151Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-dwrfw [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.919319151Z insights-runtime-extractor-5czbw [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.919319151Z insights-runtime-extractor-dfl4p [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.919319151Z insights-runtime-extractor-q9qkd [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.923012665Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.938538762Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:16.953576559Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:17.272060348Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:17.285490694Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:17.645948239Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:17.696779395Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:17.801213827Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:17.812365684Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:17.821531557Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:17.821531557Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:17.866437479Z No resources found [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:17.989371521Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:18.115058319Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:18.115196593Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:18.274277307Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:18.484879556Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:18.576146333Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:18.760487991Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:18.786154360Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.332501034Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.336777352Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.434740517Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.438310430Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.488443400Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.492525802Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.520783579Z No resources found [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.611740703Z INFO: 'previous-logs' folder not found on ip-10-0-128-42.ec2.internal, skipping... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.616273389Z tar: Removing leading `/' from member names [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.625064744Z INFO: HAProxy config collection complete. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.716163198Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.757242079Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:19.770918889Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.092090619Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.138525599Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.181374040Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.187632881Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.217620903Z INFO: 'previous-logs' folder not found on ip-10-0-132-130.ec2.internal, skipping... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.259606813Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.399045492Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.543549630Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.591461620Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.594568237Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.598150131Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.673036017Z INFO: 'previous-logs' folder not found on ip-10-0-133-160.ec2.internal, skipping... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.673195377Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.673264093Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.940952711Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:20.971754760Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.044315592Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.048402292Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.079878652Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.200445229Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.307231913Z W0716 18:28:21.307184 1279 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.362054221Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.389734163Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.395797662Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.465978926Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.471147171Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.533477728Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.537753524Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.644173486Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.753212851Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.808414591Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.890477209Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.960991060Z INFO: INTERCONNECT MODE [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:21.961038368Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.103994800Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.160806789Z INFO: Gathering OVN_Northbound from ovnkube-node-7wsjz... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.161029171Z INFO: Gathering OVN_Northbound from ovnkube-node-h7t4d... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.161316907Z INFO: Gathering OVN_Northbound from ovnkube-node-xvjmz... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.161592523Z INFO: Gathering OVN_Southbound from ovnkube-node-7wsjz... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.162056340Z INFO: Gathering OVN_Southbound from ovnkube-node-h7t4d... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.163521013Z INFO: Gathering OVN_Southbound from ovnkube-node-xvjmz... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.221783355Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.528770915Z tar: Removing leading `/' from member names [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.530251123Z tar: Removing leading `/' from member names [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.539970542Z tar: Removing leading `/' from member names [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.591092884Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.595038039Z tar: Removing leading `/' from member names [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.606729746Z tar: Removing leading `/' from member names [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.625945312Z tar: Removing leading `/' from member names [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.888198319Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:22.918451435Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.290773395Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.298764207Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.315813016Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.316976597Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.318410401Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.320902000Z 4.9M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.324563495Z ovnk_database_store/ [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.326046997Z ovnk_database_store/ovnkube-node-7wsjz_nbdb [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.346301498Z ovnk_database_store/ovnkube-node-xvjmz_nbdb [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.358456005Z ovnk_database_store/ovnkube-node-xvjmz_sbdb [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.401434866Z ovnk_database_store/ovnkube-node-h7t4d_sbdb [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.439538676Z ovnk_database_store/ovnkube-node-h7t4d_nbdb [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.452992171Z ovnk_database_store/ovnkube-node-7wsjz_sbdb [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.495235933Z INFO: Network log collection complete. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.566288716Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.653420040Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:23.655772017Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:24.184859317Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:24.274740493Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:24.386541919Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:24.413267975Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:24.532640612Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:24.660800168Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:24.665585619Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:24.712946788Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:24.848317797Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:24.940657907Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:25.209054469Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:25.211814643Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:25.361383457Z Gathering data for ns/openshift-console... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:25.408771563Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:25.645536692Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:25.651461638Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:25.844299548Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:25.866985196Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:26.090673801Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:26.094968936Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:26.110352476Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:26.310238295Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:26.541371519Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:26.544741783Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:26.761258295Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:26.769928864Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:27.023269847Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:27.028757882Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:27.037364913Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:27.196181386Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:27.244552898Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:27.439154485Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:27.441416462Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:27.676754831Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:27.939816615Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:27.943065053Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:27.967125254Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:28.140845406Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:28.182896471Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:28.296348341Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:28.376393433Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:28.379584743Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:28.581958577Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:28.628364646Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:28.822069644Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:28.824587245Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:29.024122545Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:29.159358858Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:29.265426128Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:29.268439782Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:29.322281293Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:29.449478024Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:29.649940082Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:29.652275009Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:29.858692816Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:30.084927024Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:30.089468222Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:30.250436986Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:30.286125703Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:30.489411218Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:30.533919306Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:30.536778884Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:30.751180723Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:30.857667810Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:30.975943197Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:30.978691925Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:31.170124883Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:31.399581122Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:31.402572775Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:31.618529479Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:31.662186521Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:31.863223822Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:31.866917401Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:32.081956850Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:32.299942756Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:32.302712012Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:32.488527303Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:32.701238959Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:32.703743835Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:32.808350400Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:32.905741991Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:33.113745544Z Collecting performance related data for node ip-10-0-132-130.ec2.internal [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:33.122002708Z Collecting performance related data for node ip-10-0-133-160.ec2.internal [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:33.144085187Z Collecting performance related data for node ip-10-0-128-42.ec2.internal [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:33.159381492Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:33.162643268Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:33.302175216Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:33.413471936Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:33.689278271Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:33.861981787Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:34.129954574Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:34.652618302Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:34.936903982Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:35.120229558Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:35.271413196Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:35.420841478Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:35.538319146Z Gathering data for ns/kserve... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:35.634475755Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:36.656632491Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:36.921323299Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:37.087011178Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:37.617455015Z Done executing Istio gather script [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:38.010585811Z Gathering data for ns/openshift-config... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:38.307650136Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:38.364579175Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:38.953050667Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:39.322234207Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:39.675056954Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:40.052162340Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:40.434472457Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:40.795067241Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:41.201324666Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:41.700285961Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:41.763222219Z Collecting kubelet logs for node ip-10-0-132-130.ec2.internal [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:41.768616559Z Collecting kubelet logs for node ip-10-0-133-160.ec2.internal [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:41.773190139Z Collecting kubelet logs for node ip-10-0-128-42.ec2.internal [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:42.170817875Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:42.202655154Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-qh77t namespace [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:42.209396585Z INFO: Node performance data collection complete. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:42.817431575Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:43.312755662Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:44.488398825Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:45.882017548Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:46.275653205Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:46.892186789Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:47.425730480Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:47.888159433Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:48.252145456Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:48.317692280Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:48.821762890Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:49.204222835Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:49.699172646Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:50.047168192Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:50.388135979Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:51.157544335Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:51.515067216Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:51.899372094Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:52.398826281Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:53.322417160Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:53.503256994Z Wrote inspect data to must-gather. [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:53.503303245Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:53.503303245Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:53.643480672Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-82ktg] POD 2026-07-16T18:28:53.952895081Z Caches written to disk [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:54.964816918Z waiting for gather to complete [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:54.968349358Z downloading gather output [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.228112167Z receiving incremental file list [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.248963015Z ./ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.249056131Z aggregated-discovery-api.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.24936Z aggregated-discovery-apis.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.250131111Z event-filter.html [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.253080953Z timestamp [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.253156338Z version [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.260091319Z cluster-scoped-resources/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.260121121Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.260127412Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.260157034Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.260296383Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.260428061Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.260501256Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.26056153Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.260678988Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.260780814Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.260923244Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.260938875Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.261083944Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.261149168Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.261598388Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.261658301Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.261875496Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.262125972Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.262374178Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.262580072Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.266627305Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.268177896Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.268319566Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.268459835Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.268577162Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.268719292Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.269195383Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.269341312Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.269509313Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.269709126Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.270508888Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.270781616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.270906984Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.271092486Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.272310825Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.272575503Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.272744143Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.272904774Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.273220385Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.273347803Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.273580838Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.273855366Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.274585663Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.274743484Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.274886003Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.275002261Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.275103477Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.275217615Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.275373855Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.275584068Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.276573893Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.276716772Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.276851211Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.278847531Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.279011142Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.279168602Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.279431809Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.279713417Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.280139665Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.280266433Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.280376171Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.280533221Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.280644248Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.280764256Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.280882113Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.281066876Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.281201934Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.281335543Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.281664995Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.281798933Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.281911681Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.282489338Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.282618507Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.282840031Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.283045854Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.283959494Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.284114214Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.28435451Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.284526421Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.284752466Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.284932367Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.28513069Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.285306862Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.285487104Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.285644044Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.285804854Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.285953374Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.286124795Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.286319078Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.286527491Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.286724724Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.286899505Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.287342674Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.28758859Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.288491179Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.288639859Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.289586731Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.289757762Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.289967185Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.290062871Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.290222862Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.290539843Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.290850683Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.291010323Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.291151953Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.291347645Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.291527617Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.291675807Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.291854908Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.292022429Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.296602327Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.297148803Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.297848569Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.298073783Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.298277366Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.298475259Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.298659731Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.298809571Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.298965761Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.299929994Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.30032825Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.303893892Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.304056873Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.304240775Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.304423207Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.304586168Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.304728667Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.304885967Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.307950307Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.311484057Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.316720238Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.321144846Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.321288645Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.321627827Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.321778257Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.321918766Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.322019833Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.322175783Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.322457481Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.32259304Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.322700537Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.322885059Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.323013328Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.323131565Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.323235062Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.32335085Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.32350531Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.323649189Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.323757416Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.323863363Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.323985231Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.324143731Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.324241157Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.324381097Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.324520986Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.324650024Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.324936243Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.325063391Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.325303447Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.325411564Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.325559454Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.32565616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.327782008Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.327934888Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.328023604Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.32812089Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.328224947Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.328372517Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.328509436Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.328661766Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.328782704Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.330367167Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.330551248Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.330669106Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.330814616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.330927633Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.331105485Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.331347991Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.331864694Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.332129731Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.332245419Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.332362377Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.332487215Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.333108135Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.333297398Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.333415425Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.334857009Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.335004329Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.335149888Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.335252775Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.335457528Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.335739257Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.335856404Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.336012374Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.336426521Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.336549009Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.336674847Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.336915373Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.337074444Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.337205952Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.337344881Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.337539624Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.337832993Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.337898017Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.337989083Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.338073558Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.338157244Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.338236319Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.338384489Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.338528788Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.338612394Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.338695179Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.338779175Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.338874151Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.338992878Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.339073474Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.339200102Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.339292058Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.339374963Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.339487121Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.339569046Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.339655362Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.339743017Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.339830973Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.339912968Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.339998274Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.34009541Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.340172135Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.340259381Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.340348797Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.340478545Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.340576632Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.340669508Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.340748993Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.340842309Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.340933105Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.341027971Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.341115057Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.341210323Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.341288818Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.341385174Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.341504282Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.341584637Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.341667763Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.341755468Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.341840234Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.34193068Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.342013875Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.342103121Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.342193047Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.342314665Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.342415281Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.34254802Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.342633175Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.342722611Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.342816698Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.342915854Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.3430031Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.343091575Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.343185251Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.343272987Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.343362653Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.343481041Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.343566566Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.343667993Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.343756089Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.343856125Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.343967182Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.344061648Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.344156725Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.344244171Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.344337886Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.344457014Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.34454422Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.344638086Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.344772705Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.344914764Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.345016051Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.345116027Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.345208733Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.34531333Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.345421627Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.345541505Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.34562825Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.345729987Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.345830654Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.34592234Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.346020066Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.346114462Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.346208368Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.346300074Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.346412592Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.346574722Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.346672169Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.346800517Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.346898383Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.346986479Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.347082275Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.347172271Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.347269528Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.347358963Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.347479131Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.347567497Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.347657493Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.347747239Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.347839785Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.347928731Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.348023117Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.348111242Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.348262992Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.348313245Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.348498648Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.348639387Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.348716372Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.348801027Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.349131069Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.349235776Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.349336742Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.34945019Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.349558557Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.349642702Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.349728918Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.349814603Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.349901799Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.349986394Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.350081751Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.350166436Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.350261883Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.350351088Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.350466426Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.350566512Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.350641197Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.350726023Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.350806228Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.350849101Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.350950687Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.351042043Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.351129409Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.351259717Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.351354163Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.351475751Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.351630112Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.351719607Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.351822024Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.35191375Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.352009886Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.352141515Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.352268343Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.35237473Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.352514099Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.352612436Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.352703691Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.352798898Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.352888663Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.35298938Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353062255Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353109098Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353170332Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353221915Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353279549Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353343083Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353425089Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353540246Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.35359516Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353636282Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353690496Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353739999Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353802483Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353810223Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353836215Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.353946012Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.354037198Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.354100313Z cluster-scoped-resources/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.354109603Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.354168387Z cluster-scoped-resources/core/nodes/ip-10-0-128-42.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.354328607Z cluster-scoped-resources/core/nodes/ip-10-0-132-130.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.354502909Z cluster-scoped-resources/core/nodes/ip-10-0-133-160.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.354624336Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.354686921Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.354734234Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.35483229Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.354918786Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.355006561Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.355091977Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.355173602Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.355315392Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.355408298Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.355522775Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.355609631Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.355700497Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.355794633Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.355880008Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.355969674Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.3560612Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.356146396Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.356236452Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.356322257Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.356436025Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.35652701Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.356612516Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.356694491Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.356731744Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.356836951Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.356913056Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.356996481Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357076706Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357161002Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357299641Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.35744884Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357547067Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357610391Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357619672Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357674905Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357729869Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357736589Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357762771Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357848526Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.35789705Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357972755Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.357987716Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.358073121Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.358195109Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.358374151Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.358471027Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.358505929Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.358536641Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.358634848Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.358646069Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.358684701Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.358766816Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.358804629Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.35896849Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359001892Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359062405Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359103308Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359167932Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359183724Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.35959582Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359629633Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359681006Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359723939Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359763821Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359823415Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359869248Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359910331Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.359970975Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360020108Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360088543Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360136055Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36020176Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360236332Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360286805Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360333028Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360383002Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360451576Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360551063Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360558943Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360607576Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360692842Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360734705Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360843902Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.360936198Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.361088818Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.361193564Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.361294991Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36142717Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36159081Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.361649654Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.361699457Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.361779593Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.361860278Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.362000147Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.362121525Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36220553Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.362299996Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.362409763Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36251409Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.362595266Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.362697282Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.362777767Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.362885154Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36296701Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.363048305Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.363149512Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.363232137Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.363321063Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.363417349Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.363531197Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.363625113Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36374349Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.363789313Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36388748Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.363970135Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.364065831Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36419896Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.364290536Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.364383672Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36450312Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.364588756Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.364675521Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.364762207Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.364859693Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.364942609Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.365032874Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36512161Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.365207256Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.365299132Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.365413069Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.365503585Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.365602582Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.365699048Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.365782943Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.365865169Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.365953954Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36604322Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366136196Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366230132Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366314118Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366425045Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366539592Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366599196Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366607427Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36665258Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366712104Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366726225Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366771318Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36681413Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366824761Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366875834Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366932638Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.366981191Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-128-42.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367067187Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-132-130.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367152983Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-133-160.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367187355Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367238218Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367322254Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367369737Z host_service_logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367446782Z host_service_logs/masters/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367491725Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367560719Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367635044Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367699098Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367859189Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.367988847Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.368061592Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.368136747Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.368202781Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.368276646Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36834193Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.368411985Z ingress_controllers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.368425386Z ingress_controllers/default/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.368429416Z ingress_controllers/default/router-default-568468f9f4-9h48n/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.368462648Z ingress_controllers/default/router-default-568468f9f4-9h48n/haproxy.config [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.368621058Z insights-data/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36865117Z istio/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.368661491Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.36880289Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.369317404Z istio/event-filter.html [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.369589481Z istio/timestamp [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.369706419Z istio/cluster-scoped-resources/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.369716099Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.3697203Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.369778563Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.369853418Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.369906902Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.369980477Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.370064592Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.370114615Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.370292337Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.371125041Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.371352156Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.371527097Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.371919563Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.372090794Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.37249578Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.373321424Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.373483155Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.373606113Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.373732721Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.373831707Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.373942745Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.374101005Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.374269176Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.374728226Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.374923779Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.375315624Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.375465224Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.375613414Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.375744752Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.375753783Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.375788215Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.37586264Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.37587085Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.375914033Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376005479Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376144288Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376207602Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376259306Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376361232Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376548964Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376614479Z istio/namespaces/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376624769Z istio/namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.37663849Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376701805Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376755948Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376856835Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.376910358Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/fc-concurrency-test-kserve-route.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.377023886Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/fc-smoke-test-kserve-route.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.377132133Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.377307324Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-router-managed-test-llm-4b931143-kserve-route.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.377523118Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.377598893Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.377627755Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.377680788Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/fc-concurrency-test-inference-pool.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.377775484Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/fc-smoke-test-inference-pool.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.37786588Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.37801128Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-router-managed-test-llm-4b931143-inference-pool.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.378105136Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-router-managed-test-llm-5b1e8f15-inference-pool.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.37816831Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.378211383Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.378227224Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/fc-concurrency-test-inference-pool.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.378334011Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/fc-smoke-test-inference-pool.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.378438278Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.378594158Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-router-managed-test-llm-4b931143-inference-pool.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.378684894Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-router-managed-test-llm-5b1e8f15-inference-pool.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.378749998Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.378798431Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.378836713Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/fc-concurrency-test-kserve-scheduler.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.378931Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/fc-concurrency-test-kserve-shadow-svc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.379022306Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/fc-concurrency-test-kserve-workload-svc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.379167695Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/fc-smoke-test-kserve-scheduler.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.379255801Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/fc-smoke-test-kserve-shadow-svc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.379347377Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/fc-smoke-test-kserve-workload-svc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.379469305Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.379559231Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.379648766Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.379740983Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-4b931143-kserve-scheduler.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.379835709Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.379918984Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.380023121Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-5b1e8f15-kserve-scheduler.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.380106356Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.380195812Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.380269417Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.38032309Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.380390495Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.38046618Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.380540935Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.380613489Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.380691634Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.381024336Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.381232509Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.381269532Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.381303974Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.38139066Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.381461864Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.38154402Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.381600444Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.381647867Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.381727632Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.381784845Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.381813997Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.382289289Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.382390485Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.382650622Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.382725637Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.382980223Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.383091861Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.383301054Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.383377069Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.383455874Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.383534679Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.38354415Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.383595654Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.383670168Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.383738593Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.383803807Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.38385587Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.383933605Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384014021Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384057084Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384103667Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384173861Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384183732Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384239445Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384339612Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.38462186Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384728467Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384757029Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384810953Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384824714Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384873327Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/istiod-openshift-gateway-94bb8fbfd-qnh6j.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384982244Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384989754Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.384996055Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.385047178Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.393954538Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.394022873Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.394070166Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.394122069Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.394241027Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.394251348Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.394256478Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.394300001Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396092317Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396179123Z istio/namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396216816Z istio/namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router-default-568468f9f4-9h48n.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396338564Z istio/namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396347374Z istio/namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396352324Z istio/namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396415378Z istio/namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396533696Z istio/namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396604251Z istio/namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396649794Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396694327Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.39675414Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396800523Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396920821Z machine_config_ondisk/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396929642Z machine_config_termination_logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396936743Z monitoring/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396940633Z monitoring/alertmanager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.396983155Z monitoring/alertmanager/status.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.397104903Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.397174788Z monitoring/prometheus/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.397227151Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.397325728Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.397388152Z monitoring/prometheus/rules.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.398057465Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.398106299Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.398160042Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.39904662Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.399065291Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.399109904Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.39919549Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.399268304Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.39935372Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.399384242Z monitoring/prometheus/status/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.399436715Z monitoring/prometheus/status/config.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.399889335Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.399949169Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.400042995Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.400245148Z namespaces/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.400254268Z namespaces/cert-manager-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.400259279Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.400264479Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.400284681Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.400358205Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.400373916Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.400452441Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.400655235Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.400879559Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.401022589Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.401239453Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.401550683Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.401597876Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-96tjj.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.401752736Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.401796619Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.401850112Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.401890605Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.40196197Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.402005553Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.402071587Z namespaces/cert-manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.402080607Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.402085658Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.402101309Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.402259149Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.402457942Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.402611702Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.402809005Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403049281Z namespaces/default/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403099364Z namespaces/default/default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403164248Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403202211Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403281826Z namespaces/default/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403310678Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403421845Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.40350229Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403586436Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403642879Z namespaces/default/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403677011Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403750946Z namespaces/default/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403773528Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403883425Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403947009Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.403979801Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.404066917Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.404146772Z namespaces/default/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.404187455Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.404309403Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.404387398Z namespaces/default/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.40458008Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.404652845Z namespaces/default/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.40472899Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.404853478Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.404997607Z namespaces/default/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405109085Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405150467Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405222512Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405259294Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405303517Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.40534517Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405444397Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405530772Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405564724Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405607177Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.40565234Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405696803Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405746316Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405753847Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405786899Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.405929228Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.40610809Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406243129Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406457703Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406661946Z namespaces/default/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406697758Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406745541Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406803185Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406851598Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406861449Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406866439Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406904842Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406973556Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.406983917Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.407019299Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.407129536Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.40718002Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.407216842Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.407303798Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.407364902Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.407375012Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.407463608Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.407608758Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.407777969Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.407920658Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.408121101Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.408351936Z namespaces/kserve/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.40841248Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.408512736Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.408533708Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.408612613Z namespaces/kserve/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.408663206Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.408749982Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.408952175Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409114005Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409156108Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409203791Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409261815Z namespaces/kserve/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409319739Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409460138Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409544013Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409583246Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409688423Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409757818Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409768548Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.409849964Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.40994946Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.410026355Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.410147283Z namespaces/kserve/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.410201576Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.410463073Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.410596522Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.410892001Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.410966086Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.411259705Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.411449158Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.411700374Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.41179434Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.411830322Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.41195039Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.411981282Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412050917Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412085409Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412174405Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.41225035Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412309544Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412357737Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412431412Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412478615Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412572801Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412614473Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412651586Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.41271009Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.41271958Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412767014Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.412919533Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.413103705Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.413284427Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.413510442Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.413713075Z namespaces/kserve/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.413719925Z namespaces/kserve/pods/kserve-controller-manager-5c5755fdcb-g2x5r/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.413769059Z namespaces/kserve/pods/kserve-controller-manager-5c5755fdcb-g2x5r/kserve-controller-manager-5c5755fdcb-g2x5r.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.413870555Z namespaces/kserve/pods/kserve-controller-manager-5c5755fdcb-g2x5r/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.413879306Z namespaces/kserve/pods/kserve-controller-manager-5c5755fdcb-g2x5r/manager/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.413884226Z namespaces/kserve/pods/kserve-controller-manager-5c5755fdcb-g2x5r/manager/manager/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.413922319Z namespaces/kserve/pods/kserve-controller-manager-5c5755fdcb-g2x5r/manager/manager/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.414057678Z namespaces/kserve/pods/kserve-controller-manager-5c5755fdcb-g2x5r/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.414125732Z namespaces/kserve/pods/kserve-controller-manager-5c5755fdcb-g2x5r/manager/manager/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.414172505Z namespaces/kserve/pods/llmisvc-controller-manager-565cf7448c-6d7tb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.414215168Z namespaces/kserve/pods/llmisvc-controller-manager-565cf7448c-6d7tb/llmisvc-controller-manager-565cf7448c-6d7tb.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.414305514Z namespaces/kserve/pods/llmisvc-controller-manager-565cf7448c-6d7tb/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.414316964Z namespaces/kserve/pods/llmisvc-controller-manager-565cf7448c-6d7tb/manager/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.414340956Z namespaces/kserve/pods/llmisvc-controller-manager-565cf7448c-6d7tb/manager/manager/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.414372818Z namespaces/kserve/pods/llmisvc-controller-manager-565cf7448c-6d7tb/manager/manager/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.431724168Z namespaces/kserve/pods/llmisvc-controller-manager-565cf7448c-6d7tb/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.431769071Z namespaces/kserve/pods/llmisvc-controller-manager-565cf7448c-6d7tb/manager/manager/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.431830755Z namespaces/kserve/pods/model-serving-api-fd65d7d6-h2w48/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.431856957Z namespaces/kserve/pods/model-serving-api-fd65d7d6-h2w48/model-serving-api-fd65d7d6-h2w48.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.431972384Z namespaces/kserve/pods/model-serving-api-fd65d7d6-h2w48/server/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.431982315Z namespaces/kserve/pods/model-serving-api-fd65d7d6-h2w48/server/server/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.431992286Z namespaces/kserve/pods/model-serving-api-fd65d7d6-h2w48/server/server/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.432042279Z namespaces/kserve/pods/model-serving-api-fd65d7d6-h2w48/server/server/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.432126374Z namespaces/kserve/pods/model-serving-api-fd65d7d6-h2w48/server/server/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.432196709Z namespaces/kserve/pods/model-serving-api-fd65d7d6-h2w48/server/server/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.432240492Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-lf6xh/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.432296075Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-lf6xh/odh-model-controller-84b54b7f97-lf6xh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.432408383Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-lf6xh/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.432438995Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-lf6xh/manager/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.432446255Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-lf6xh/manager/manager/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.432483648Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-lf6xh/manager/manager/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.43743646Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-lf6xh/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437503774Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-lf6xh/manager/manager/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437532156Z namespaces/kserve/pods/s3-init-54lxp/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437578199Z namespaces/kserve/pods/s3-init-54lxp/s3-init-54lxp.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437668305Z namespaces/kserve/pods/s3-init-54lxp/download-hf-model/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437675236Z namespaces/kserve/pods/s3-init-54lxp/download-hf-model/download-hf-model/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437679296Z namespaces/kserve/pods/s3-init-54lxp/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437713368Z namespaces/kserve/pods/s3-init-54lxp/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437797614Z namespaces/kserve/pods/s3-init-54lxp/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437861568Z namespaces/kserve/pods/s3-init-54lxp/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.43789713Z namespaces/kserve/pods/s3-init-54lxp/s3-init/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437903951Z namespaces/kserve/pods/s3-init-54lxp/s3-init/s3-init/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437912931Z namespaces/kserve/pods/s3-init-54lxp/s3-init/s3-init/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.437966905Z namespaces/kserve/pods/s3-init-54lxp/s3-init/s3-init/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.438295086Z namespaces/kserve/pods/s3-init-54lxp/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.438367111Z namespaces/kserve/pods/s3-init-54lxp/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.438422754Z namespaces/kserve/pods/seaweedfs-64568bcd49-bqk2x/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.438471778Z namespaces/kserve/pods/seaweedfs-64568bcd49-bqk2x/seaweedfs-64568bcd49-bqk2x.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.438557403Z namespaces/kserve/pods/seaweedfs-64568bcd49-bqk2x/seaweedfs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.438571114Z namespaces/kserve/pods/seaweedfs-64568bcd49-bqk2x/seaweedfs/seaweedfs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.438577975Z namespaces/kserve/pods/seaweedfs-64568bcd49-bqk2x/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.438627068Z namespaces/kserve/pods/seaweedfs-64568bcd49-bqk2x/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.43881974Z namespaces/kserve/pods/seaweedfs-64568bcd49-bqk2x/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.438888755Z namespaces/kserve/pods/seaweedfs-64568bcd49-bqk2x/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.43897501Z namespaces/kserve/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.439014923Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.439087508Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.43912013Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.439208036Z namespaces/kuadrant-system/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.439254659Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.439325643Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.439367636Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.439445171Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.439479133Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.439564059Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.439874409Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.440148697Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.44020555Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.440238393Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.440314618Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.44035449Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.440464458Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.44050428Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.440559664Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.440623518Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.440675591Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.440684332Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.440714924Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.440801689Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.440888355Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.44096632Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.441015443Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.441057686Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.441256499Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.441364826Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.441938643Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.442016468Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.44234377Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.44249703Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.442709104Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.442835952Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.442855023Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.442991582Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443016444Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443080178Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443126411Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443213837Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443289171Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443328584Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443375037Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443449672Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443487174Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443558129Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443567909Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443611352Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.443818696Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.444023939Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.444171219Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.444841452Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.444855193Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.444860254Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-xxj9m.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.44526501Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.445284772Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.445302493Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.445442792Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.445750942Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.445766113Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.445770983Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.445775163Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.445779014Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.445860719Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.445995288Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.446091714Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.44618532Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.446199171Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.446223562Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino-7c77957575-xc96s.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.446305468Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.446314079Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/authorino/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.446326139Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/authorino/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.446364982Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/authorino/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.449468234Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.449541689Z namespaces/kuadrant-system/pods/authorino-7c77957575-xc96s/authorino/authorino/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.449587782Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.449638815Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/authorino-operator-6d85f6564-c4d5z.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.449734501Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.449748312Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.449760543Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/manager/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.449794665Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/manager/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.449971687Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450037071Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-c4d5z/manager/manager/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450089024Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450137828Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/dns-operator-controller-manager-65b49595d7-m7vkn.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450225123Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450238904Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450248265Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/manager/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450284907Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/manager/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450390014Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450505481Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-m7vkn/manager/manager/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450530873Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450580236Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin-85fd4b8b9d-m6wrp.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450663962Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450672912Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450677463Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450713335Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450783819Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450853214Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-m6wrp/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450877366Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.450922699Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.451034356Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.451044627Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.451049057Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/manager/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.451074238Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/manager/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.454499981Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.454574606Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-fp8pd/manager/manager/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.454616049Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.454660562Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador-limitador-69574b596d-nzkh5.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.454902518Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.454914228Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/limitador/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.454919959Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/limitador/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.454969972Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/limitador/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455050617Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455131913Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nzkh5/limitador/limitador/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455175645Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455212008Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/limitador-operator-controller-manager-6f9f468797-d7wdw.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455313945Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455321785Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455329475Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/manager/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455370738Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/manager/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455568341Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455636306Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-d7wdw/manager/manager/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455668108Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.45570727Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455764554Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455782715Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455835039Z namespaces/kube-node-lease/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455845509Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.45585803Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.455915614Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-128-42.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.45600533Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-132-130.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.456098016Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-133-160.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.456123897Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.456143589Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.456208243Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.456445668Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.456644881Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.45677979Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.456975903Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.457199217Z namespaces/kube-public/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.457210578Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.457219499Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.457255271Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.457428632Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.457604484Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.457738763Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.457948966Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.458203573Z namespaces/kube-system/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.458244065Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.458329681Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.458351792Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.458430188Z namespaces/kube-system/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.45847139Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.45861741Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.458687794Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.45877089Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.458814242Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.458845044Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.458887347Z namespaces/kube-system/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.458921769Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.459008555Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.459060878Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.45909076Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.459177216Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.459218169Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.45924206Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.459283593Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-624mm5lrvr3svwb4anjxvb4nna.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.459375569Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.459555431Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.459637646Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.459783456Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.459860451Z namespaces/kube-system/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.459901653Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.460075735Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.4601602Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.460451229Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.460527084Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.460800502Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.46091939Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461219829Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461293204Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461335657Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461417302Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461461665Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461500157Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.4615377Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461629026Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461704941Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461752674Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461780746Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461836939Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461869881Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461924285Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461943976Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.461989869Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.46214917Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.462327931Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.462490172Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.462681574Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.462897679Z namespaces/kube-system/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.462909829Z namespaces/kube-system/pods/global-pull-secret-syncer-p4pp7/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.462957403Z namespaces/kube-system/pods/global-pull-secret-syncer-p4pp7/global-pull-secret-syncer-p4pp7.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.463037628Z namespaces/kube-system/pods/global-pull-secret-syncer-p4pp7/global-pull-secret-syncer/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.463063169Z namespaces/kube-system/pods/global-pull-secret-syncer-p4pp7/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.46306796Z namespaces/kube-system/pods/global-pull-secret-syncer-p4pp7/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.463121953Z namespaces/kube-system/pods/global-pull-secret-syncer-p4pp7/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.463427043Z namespaces/kube-system/pods/global-pull-secret-syncer-p4pp7/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.463499998Z namespaces/kube-system/pods/global-pull-secret-syncer-p4pp7/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.46354013Z namespaces/kube-system/pods/global-pull-secret-syncer-rg5vf/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.463582283Z namespaces/kube-system/pods/global-pull-secret-syncer-rg5vf/global-pull-secret-syncer-rg5vf.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.463821359Z namespaces/kube-system/pods/global-pull-secret-syncer-rg5vf/global-pull-secret-syncer/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.463831139Z namespaces/kube-system/pods/global-pull-secret-syncer-rg5vf/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.4638369Z namespaces/kube-system/pods/global-pull-secret-syncer-rg5vf/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.46384352Z namespaces/kube-system/pods/global-pull-secret-syncer-rg5vf/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464134719Z namespaces/kube-system/pods/global-pull-secret-syncer-rg5vf/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464202644Z namespaces/kube-system/pods/global-pull-secret-syncer-rg5vf/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464235576Z namespaces/kube-system/pods/global-pull-secret-syncer-rp2zv/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464275688Z namespaces/kube-system/pods/global-pull-secret-syncer-rp2zv/global-pull-secret-syncer-rp2zv.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464355474Z namespaces/kube-system/pods/global-pull-secret-syncer-rp2zv/global-pull-secret-syncer/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464363644Z namespaces/kube-system/pods/global-pull-secret-syncer-rp2zv/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464374465Z namespaces/kube-system/pods/global-pull-secret-syncer-rp2zv/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464437259Z namespaces/kube-system/pods/global-pull-secret-syncer-rp2zv/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464725688Z namespaces/kube-system/pods/global-pull-secret-syncer-rp2zv/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464788572Z namespaces/kube-system/pods/global-pull-secret-syncer-rp2zv/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464848856Z namespaces/kube-system/pods/konnectivity-agent-fjqs9/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464892198Z namespaces/kube-system/pods/konnectivity-agent-fjqs9/konnectivity-agent-fjqs9.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464980214Z namespaces/kube-system/pods/konnectivity-agent-fjqs9/konnectivity-agent/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464989235Z namespaces/kube-system/pods/konnectivity-agent-fjqs9/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.464994205Z namespaces/kube-system/pods/konnectivity-agent-fjqs9/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.465037028Z namespaces/kube-system/pods/konnectivity-agent-fjqs9/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.465920505Z namespaces/kube-system/pods/konnectivity-agent-fjqs9/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.46598627Z namespaces/kube-system/pods/konnectivity-agent-fjqs9/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.466056404Z namespaces/kube-system/pods/konnectivity-agent-kq8j9/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.466097817Z namespaces/kube-system/pods/konnectivity-agent-kq8j9/konnectivity-agent-kq8j9.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.466174512Z namespaces/kube-system/pods/konnectivity-agent-kq8j9/konnectivity-agent/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.466181492Z namespaces/kube-system/pods/konnectivity-agent-kq8j9/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.466204374Z namespaces/kube-system/pods/konnectivity-agent-kq8j9/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.466264738Z namespaces/kube-system/pods/konnectivity-agent-kq8j9/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.467105843Z namespaces/kube-system/pods/konnectivity-agent-kq8j9/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.467169507Z namespaces/kube-system/pods/konnectivity-agent-kq8j9/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.467198919Z namespaces/kube-system/pods/konnectivity-agent-svjcb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.467243452Z namespaces/kube-system/pods/konnectivity-agent-svjcb/konnectivity-agent-svjcb.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.467330697Z namespaces/kube-system/pods/konnectivity-agent-svjcb/konnectivity-agent/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.467339748Z namespaces/kube-system/pods/konnectivity-agent-svjcb/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.467344668Z namespaces/kube-system/pods/konnectivity-agent-svjcb/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.467388121Z namespaces/kube-system/pods/konnectivity-agent-svjcb/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468218675Z namespaces/kube-system/pods/konnectivity-agent-svjcb/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.46828597Z namespaces/kube-system/pods/konnectivity-agent-svjcb/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468343593Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-42.ec2.internal/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468376985Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-42.ec2.internal/kube-apiserver-proxy-ip-10-0-128-42.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468489463Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-42.ec2.internal/haproxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468505424Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-42.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468510634Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-42.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468526195Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-42.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468722558Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-42.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468786732Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-42.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468838135Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-130.ec2.internal/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468875278Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-130.ec2.internal/kube-apiserver-proxy-ip-10-0-132-130.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468953933Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-130.ec2.internal/haproxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468961414Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-130.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.468971884Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-130.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469012167Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-130.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469251892Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-130.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469319127Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-130.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469344768Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469411653Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469477277Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal/haproxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469484557Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469488668Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469529731Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.4698346Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469905795Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469923756Z namespaces/kube-system/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.469969869Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.470009142Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.470018652Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.470072216Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.470171812Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.470232046Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47028393Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.470335933Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.470372735Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.470435379Z namespaces/open-cluster-management-1843c268-292f-4bb4-8301-17216fecb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47044756Z namespaces/open-cluster-management-1843c268-292f-4bb4-8301-17216fecb/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.470458041Z namespaces/open-cluster-management-1843c268-292f-4bb4-8301-17216fecb/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.470524455Z namespaces/open-cluster-management-1843c268-292f-4bb4-8301-17216fecb/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.470715368Z namespaces/open-cluster-management-1843c268-292f-4bb4-8301-17216fecb/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47090236Z namespaces/open-cluster-management-1843c268-292f-4bb4-8301-17216fecb/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.471043469Z namespaces/open-cluster-management-1843c268-292f-4bb4-8301-17216fecb/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.471251793Z namespaces/open-cluster-management-1843c268-292f-4bb4-8301-17216fecb/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.471467027Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.471477837Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.471481738Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.471508289Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.471608496Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.471651339Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.471661559Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.471714573Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.471882194Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.472084787Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.472245097Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.472498344Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47274117Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.472786792Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.472875778Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.472920721Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.472995406Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473022808Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473102523Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473197959Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473278845Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473326648Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47336016Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473419864Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473473967Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473564073Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473598565Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473641238Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473724364Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473780857Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.473824Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.474866288Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.474947193Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.475030279Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.475113474Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.475191989Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.475328478Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.475587955Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47566008Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.475710063Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.475760516Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.475805969Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.475852192Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.475889565Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.475981641Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476060966Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47611724Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476168043Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476229997Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476270499Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476320383Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476370866Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476442141Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476452911Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476489964Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476656775Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476833366Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.476968755Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477181109Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477382312Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477426765Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477495099Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477532882Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477626058Z namespaces/openshift-apiserver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477673581Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477746426Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477773748Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477830181Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477878574Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.477955079Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.478041655Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47811539Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.478167453Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.478190555Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47826642Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.478321043Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47842053Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.478656595Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.478695197Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.478785683Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.478856718Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47889185Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.479021809Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.479100914Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47918962Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.479263065Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47935018Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47949892Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.479669171Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.479702263Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.479744896Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47980866Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.479859233Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.479911027Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.47996353Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.480037495Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.48011514Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.480170384Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.480202375Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.4802778Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.480313033Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.480348385Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.480383037Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.480443901Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.480594321Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.480773873Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.480921422Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481131166Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481314348Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481354541Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481424185Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481468118Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481532052Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481539693Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481543853Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481589696Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.48165163Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481661191Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481722185Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.481869484Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.482044716Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.482191405Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.482387628Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.482600982Z namespaces/openshift-authentication/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.482612863Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.482625384Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.482672047Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.482824186Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.483002038Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.483138117Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.483532652Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.483714744Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.483722045Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.483728935Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.483771068Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.483841873Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.483851563Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.483899196Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.484050776Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.484230048Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.484364157Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.484585251Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.484779194Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.484787974Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.484792405Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.484838068Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.484898512Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.484906032Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.484948145Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.485012929Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.48502261Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.485062482Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.485213982Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.485406595Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.485547504Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.485765278Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.485975462Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486010284Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486242679Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486275671Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486351576Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486383818Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486506976Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486586181Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486666827Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486702869Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486735441Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486789515Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486846318Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486967016Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.486986337Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487067563Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487155208Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487197661Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487208322Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487262706Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487379943Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487434266Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487560215Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487650921Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487733786Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487834003Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.487930989Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488109671Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488238859Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488281742Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488353117Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488419781Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488483785Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488522717Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488596833Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488689448Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488733191Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488853589Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488923574Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.488989988Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.489032271Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.489106546Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.489136297Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.489215173Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.489240294Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.489248445Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.489421946Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.489608088Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.489740267Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.489957071Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490131542Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490160264Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490218388Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490227348Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490258981Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490318954Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490355677Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490417311Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490459964Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490583182Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490623154Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490691839Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490722141Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490799256Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490829768Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.490981548Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491058303Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491138658Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491184941Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491202822Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491272607Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.49132132Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491423166Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491537054Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491569746Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491660342Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491716956Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491725346Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491753278Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.491847864Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.49193222Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.492055278Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.492155954Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.492187476Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.492299334Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.492379049Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.492736022Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.492811217Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.49301748Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.4931645Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493276987Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.49332515Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493354762Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493413736Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493481611Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493502772Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493558085Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493606249Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.49362342Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493718406Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493795831Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493833203Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493874456Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493912039Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.493956201Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.494020206Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.494031136Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.49408553Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.494447123Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.494641926Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.494789996Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.494996069Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495237455Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495246155Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495297789Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/aws-ebs-csi-driver-node-2s4gb.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495415386Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-driver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495434768Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-driver/csi-driver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495445099Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-driver/csi-driver/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495481571Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495571637Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495641631Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495676294Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-liveness-probe/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495687664Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495700365Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495730937Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495825293Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495898368Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.49593312Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-node-driver-registrar/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495946011Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.495950411Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496006795Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496097601Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496163135Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-2s4gb/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496202728Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496249001Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/aws-ebs-csi-driver-node-7pfwh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496340267Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-driver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496350887Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-driver/csi-driver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496361888Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-driver/csi-driver/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496416972Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496608444Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496670688Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496714521Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-liveness-probe/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496725232Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496731192Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496765384Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.4968582Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.496926125Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497078115Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-node-driver-registrar/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497085755Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497090466Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497143129Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497248486Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.49731023Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-7pfwh/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497371294Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497419507Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/aws-ebs-csi-driver-node-cn7hz.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497550405Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-driver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497565546Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-driver/csi-driver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497579337Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-driver/csi-driver/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.49761226Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497705455Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497766119Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497820293Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-liveness-probe/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497829274Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497833274Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497871066Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.497961752Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498022126Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498063569Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-node-driver-registrar/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498073799Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.49808627Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498125653Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498218109Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498284693Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-cn7hz/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498342437Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498357138Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498454924Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498466475Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498492327Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498576792Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498617635Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498664808Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498712901Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498773455Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498782116Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498786286Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498828159Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498886343Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498893333Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.498934966Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.49900063Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.499012331Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.499049623Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.499200123Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.499377855Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.499548466Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.499755639Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.499959213Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.499987674Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5000714Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500105742Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500188267Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500200338Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500339137Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500425153Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500518679Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500554511Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500584063Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500718262Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500742113Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500791657Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500830019Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500921015Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.500945297Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.501006161Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.501083326Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.501122358Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.501139359Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.501166881Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.501274018Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.501343232Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.502250312Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.50237554Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.502569272Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.502640137Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5028377Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.502998831Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.503139209Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.503200984Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.503255017Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.503389536Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.503444849Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.503501503Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.503556907Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.50361316Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.503659774Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.503740729Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.503852696Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.50391681Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.503965063Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.504033418Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.50406988Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.504145185Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.504182057Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.504243052Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.504278264Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.504321517Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.504351039Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.504408542Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.504605045Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.504802008Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.504947357Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.505184053Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.50560558Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.505624831Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-2qm5d/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.505672785Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-2qm5d/tuned-2qm5d.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.505772911Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-2qm5d/tuned/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.505818994Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-2qm5d/tuned/tuned/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.505825755Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-2qm5d/tuned/tuned/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.505890649Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-2qm5d/tuned/tuned/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506001416Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-2qm5d/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5060679Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-2qm5d/tuned/tuned/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506137925Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-78ctd/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506174127Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-78ctd/tuned-78ctd.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506324697Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-78ctd/tuned/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506334228Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-78ctd/tuned/tuned/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506344658Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-78ctd/tuned/tuned/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506355639Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-78ctd/tuned/tuned/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.50652259Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-78ctd/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506591254Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-78ctd/tuned/tuned/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506645648Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spdgx/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506664959Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spdgx/tuned-spdgx.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506755515Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spdgx/tuned/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506788167Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spdgx/tuned/tuned/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506794798Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spdgx/tuned/tuned/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506809219Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spdgx/tuned/tuned/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506889394Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spdgx/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506956928Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spdgx/tuned/tuned/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.506975029Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507027203Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507086997Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507107538Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507162122Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507183973Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507227376Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-128-42.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507319482Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-132-130.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507426249Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-133-160.ec2.internal.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507492923Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507539806Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507632902Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.507666975Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508089012Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508137175Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508203789Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508242532Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508316757Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508468597Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508572934Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508610576Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508659289Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508696302Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508733224Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.50882698Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508883554Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.508927777Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.509002351Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.509077106Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.509120099Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.509300201Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.509409168Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.509546237Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.509625692Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.50974766Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.509875968Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510071171Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510114614Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510155326Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510227101Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510269434Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510337958Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510380691Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510498199Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510580554Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510613016Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510655829Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510712533Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510756246Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510851832Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510892754Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510942328Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.510986851Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.511039574Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.511047915Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.511060796Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.511230156Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.511455281Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.51159744Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.511822745Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512013078Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512026518Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512037899Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator-65db858894-msl7k.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512142646Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator-watch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512153047Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512159387Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.51220631Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512301936Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512370191Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512430885Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512446666Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512452426Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.512531901Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513179683Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513245598Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-msl7k/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513297361Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513316452Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513387537Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513447091Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513540757Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.51359261Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513658535Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513697347Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513761031Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513786543Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513878829Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.513974585Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514069961Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514121535Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514152257Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514221261Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514257384Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514341349Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514374111Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514452376Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514544502Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514578485Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514587105Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514637529Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514726124Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.51481463Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.514836801Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.51496485Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.515055826Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.515367986Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.515471483Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.515590851Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.515793324Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.515962135Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516027049Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516071842Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516133666Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516165108Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516239653Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516280405Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516345Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516385492Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.51650438Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516592256Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516644399Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516687952Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516741686Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516783758Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516873334Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516914877Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.51695714Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.516997252Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.517080428Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.517091398Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.517142272Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.517298322Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.517505945Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.517642904Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.517851548Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.51804397Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518053611Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-nqvsk/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518076822Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-nqvsk/volume-data-source-validator-78495597f6-nqvsk.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518165788Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-nqvsk/volume-data-source-validator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518177849Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-nqvsk/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518184269Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-nqvsk/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.51819541Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-nqvsk/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518290086Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-nqvsk/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518363101Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-nqvsk/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518380062Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518468038Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5185061Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518513051Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518583075Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518622848Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518675311Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518736625Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518791029Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518874054Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518917147Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.518982001Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519021814Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519063317Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519102959Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519182565Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519257599Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519335154Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5194167Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519499875Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.51957354Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519606582Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519691227Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.51973383Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519775363Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519860969Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519905761Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519916192Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.519970406Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.520044471Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.520075583Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.52018569Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.520265545Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.520359921Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.520460098Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.520539473Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.520666771Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.520847713Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.520918367Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.520969111Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521019774Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521068847Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.52111376Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521165024Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521232258Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521319974Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521343045Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.52141092Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521472594Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521505916Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521549439Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521559879Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521632564Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521770943Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.521959705Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.522098004Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.522304948Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.522531752Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.522558384Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.52264164Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.522664281Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.522771558Z namespaces/openshift-config-managed/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.52279114Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.522880205Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.522898936Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523004394Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523021424Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.52309662Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523190335Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523268131Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523307773Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523352026Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523422891Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523445752Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523542509Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523593402Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523647005Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523727301Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523802635Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.523853119Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.52694801Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527023745Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527110831Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527195356Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527263111Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527453323Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527567461Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527618904Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527664247Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527739772Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527818367Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527852899Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527899082Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.527945415Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528002489Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528032361Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528125517Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528199572Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528251535Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528289628Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528334621Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528388974Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528468719Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528490961Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528521833Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528707585Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.528916908Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.529052527Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.529350367Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.529663647Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.529688459Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.529748393Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.529760774Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.529821918Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.529870801Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.529908123Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.529974908Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.53001518Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.530113757Z namespaces/openshift-config-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.530125017Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.530132818Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.53016856Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.530231254Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.530241065Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.530287688Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.530464839Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.530665502Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.530800071Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.531002364Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.531219269Z namespaces/openshift-config/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.531259841Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.531461114Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.531509777Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.531587573Z namespaces/openshift-config/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.531607214Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.53170543Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.531781635Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.53185353Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.531912254Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.531953016Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5320164Z namespaces/openshift-config/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532052063Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532138898Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532184001Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532240295Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532310469Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532361463Z namespaces/openshift-config/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532412386Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532558706Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532639021Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532725007Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532818722Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.532905208Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533059408Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533190557Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533259511Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533283603Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533337496Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533367188Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533499117Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533537169Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533621435Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533642626Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533727792Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533803967Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.53385324Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533882622Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533947906Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.533980378Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.534053393Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.534065604Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.534101216Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.534255526Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.534454669Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.534612619Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.534818083Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535007575Z namespaces/openshift-config/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535026026Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535108452Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535128553Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535218049Z namespaces/openshift-console-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535273353Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535381829Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535446344Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535528099Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535550011Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535638216Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535756524Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535858891Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535921955Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.535976808Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.536042592Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.536086375Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.536191752Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.536258767Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.53630358Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.536413117Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.536568197Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.536576867Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.53662244Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.536716156Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.53676971Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.537412292Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5375334Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.53785146Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.537949557Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.538138899Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.538297959Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.538593569Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.538745509Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.538801602Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.538883438Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.538936051Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539026877Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539059489Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539188428Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539300625Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.53937846Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539459495Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.53953061Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539586083Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539662998Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539706921Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539770825Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539821899Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539873632Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539890493Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.539958067Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.540129259Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.540328722Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.540506283Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.540774851Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.541005166Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.541019707Z namespaces/openshift-console-operator/pods/console-operator-7866497494-drs2p/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.541038588Z namespaces/openshift-console-operator/pods/console-operator-7866497494-drs2p/console-operator-7866497494-drs2p.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.541324317Z namespaces/openshift-console-operator/pods/console-operator-7866497494-drs2p/console-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.541341198Z namespaces/openshift-console-operator/pods/console-operator-7866497494-drs2p/console-operator/console-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.541349508Z namespaces/openshift-console-operator/pods/console-operator-7866497494-drs2p/console-operator/console-operator/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.541392421Z namespaces/openshift-console-operator/pods/console-operator-7866497494-drs2p/console-operator/console-operator/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.54198419Z namespaces/openshift-console-operator/pods/console-operator-7866497494-drs2p/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.542047234Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.542103387Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.542179072Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.542220205Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.54228933Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.542310431Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.542318761Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.542388286Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5426046Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.542824814Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.542986585Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.543234641Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.543510499Z namespaces/openshift-console/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.543571883Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.54399565Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.544054104Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.54414284Z namespaces/openshift-console/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.544200404Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.54430054Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.544879058Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.545191339Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.545229101Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.545298146Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5453706Z namespaces/openshift-console/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.545538191Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.545650978Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.545712582Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.545768606Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.545885114Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.545962239Z namespaces/openshift-console/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.546013842Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.546719818Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.546827665Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.547129785Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.54721709Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.547438545Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.547602165Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.54782669Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.547911886Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.54797066Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.548065796Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.54813218Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.548222066Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.54827792Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5484372Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.548551357Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.548630243Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.548670155Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.548756661Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.548800763Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.548870178Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.548916421Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.549021998Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.549034449Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.549090093Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.549265574Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.549496859Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.54966968Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.549882724Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550093378Z namespaces/openshift-console/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550117549Z namespaces/openshift-console/pods/console-5c9855ffcc-s2jwl/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550187374Z namespaces/openshift-console/pods/console-5c9855ffcc-s2jwl/console-5c9855ffcc-s2jwl.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550308122Z namespaces/openshift-console/pods/console-5c9855ffcc-s2jwl/console/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550317982Z namespaces/openshift-console/pods/console-5c9855ffcc-s2jwl/console/console/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550322723Z namespaces/openshift-console/pods/console-5c9855ffcc-s2jwl/console/console/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550387807Z namespaces/openshift-console/pods/console-5c9855ffcc-s2jwl/console/console/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550560008Z namespaces/openshift-console/pods/console-5c9855ffcc-s2jwl/console/console/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550643814Z namespaces/openshift-console/pods/console-5c9855ffcc-s2jwl/console/console/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550697087Z namespaces/openshift-console/pods/downloads-68c9cbb497-lrlw4/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55075129Z namespaces/openshift-console/pods/downloads-68c9cbb497-lrlw4/downloads-68c9cbb497-lrlw4.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550860328Z namespaces/openshift-console/pods/downloads-68c9cbb497-lrlw4/download-server/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550870418Z namespaces/openshift-console/pods/downloads-68c9cbb497-lrlw4/download-server/download-server/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550874709Z namespaces/openshift-console/pods/downloads-68c9cbb497-lrlw4/download-server/download-server/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.550924622Z namespaces/openshift-console/pods/downloads-68c9cbb497-lrlw4/download-server/download-server/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.551309507Z namespaces/openshift-console/pods/downloads-68c9cbb497-lrlw4/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.551411813Z namespaces/openshift-console/pods/downloads-68c9cbb497-lrlw4/download-server/download-server/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55150291Z namespaces/openshift-console/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.551536012Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.551617247Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.551641288Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.551801249Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.551860553Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.551936998Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55197337Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.552050655Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.552086077Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.552185204Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55227671Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.552372606Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55243712Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.552492434Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55257666Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.552612302Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.552721879Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.552793774Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.552857258Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.552968495Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.553066341Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.553119115Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.553271685Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.553382552Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.553527241Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.553620717Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.553729284Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.553879904Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554065397Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554130971Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554175984Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554248378Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554301842Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554413679Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554625463Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554696618Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554795904Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554824836Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554912652Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554934053Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.554994487Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55504669Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.555058721Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.555096244Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.555280716Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.555533892Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.555710274Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55595788Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.556193935Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.556246459Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55626498Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.556346075Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.556476254Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.556547078Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.556614632Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.556669936Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55673024Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.556783554Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5568797Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.556981506Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.557069152Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.557136817Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.557172959Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.557260394Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.557296207Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.557420785Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.557478819Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.557535352Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55764636Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.557724815Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55780827Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.557865884Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.557983902Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.558045716Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.558184235Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.558294272Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.558410339Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.558552839Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.558651855Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.558806335Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.558986337Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.559074943Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.559118335Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.55918849Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.559224242Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.559294557Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.559329389Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.559462168Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.559560434Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.559816641Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.559851203Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.559927348Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.559974491Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.560023454Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.560033645Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.560084398Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5602633Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.560497395Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.560671137Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.560910772Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561097355Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561117676Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56118489Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561222143Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561327779Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561363992Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561463178Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56149022Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561551834Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561599137Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561680573Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561757227Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561836593Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561881166Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561932529Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.561968661Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.562005143Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56210455Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.562163374Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.562199206Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.562284332Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.562356626Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.562390509Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.562655396Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.562730141Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.562810376Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.562894361Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.562970297Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563088964Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563261416Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563338461Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563371633Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563454028Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56348164Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563536323Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563569996Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563659461Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563735536Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563771529Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563817322Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563854814Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.563906987Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.564002614Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.564041746Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56410051Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.564117991Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.564183145Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.564193136Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.564209547Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.564378128Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.564585802Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.564791375Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.564996768Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.565225953Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.565257935Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56533113Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.565354292Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.565492671Z namespaces/openshift-dns/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.565522383Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.565600588Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.565620609Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.566350607Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.566382639Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.566593693Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.566685828Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.566802016Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.566826758Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.566887582Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.566956646Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56701227Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.567107616Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.567163659Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.567220443Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.567298858Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.567372523Z namespaces/openshift-dns/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56748436Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.567597748Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.567689244Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.567921599Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.568001534Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.568283963Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.568416341Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.568600593Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.568651556Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56869988Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.568771334Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.568810377Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56885717Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.568908653Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.568991619Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.569067503Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.569133948Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.569187211Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.569231384Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.569280668Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5693246Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.569345932Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.569421737Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.569442668Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.569456869Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56962086Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.569805192Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.56992611Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.570126732Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.570563841Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.570580362Z namespaces/openshift-dns/pods/dns-default-4h2x5/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.570623225Z namespaces/openshift-dns/pods/dns-default-4h2x5/dns-default-4h2x5.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.570715501Z namespaces/openshift-dns/pods/dns-default-4h2x5/dns/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.570724891Z namespaces/openshift-dns/pods/dns-default-4h2x5/dns/dns/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.570732552Z namespaces/openshift-dns/pods/dns-default-4h2x5/dns/dns/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.570774865Z namespaces/openshift-dns/pods/dns-default-4h2x5/dns/dns/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.570889452Z namespaces/openshift-dns/pods/dns-default-4h2x5/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.570955786Z namespaces/openshift-dns/pods/dns-default-4h2x5/dns/dns/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57100287Z namespaces/openshift-dns/pods/dns-default-4h2x5/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57101619Z namespaces/openshift-dns/pods/dns-default-4h2x5/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571025951Z namespaces/openshift-dns/pods/dns-default-4h2x5/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571064104Z namespaces/openshift-dns/pods/dns-default-4h2x5/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57116596Z namespaces/openshift-dns/pods/dns-default-4h2x5/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571237195Z namespaces/openshift-dns/pods/dns-default-4h2x5/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571273007Z namespaces/openshift-dns/pods/dns-default-5dwz7/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571303999Z namespaces/openshift-dns/pods/dns-default-5dwz7/dns-default-5dwz7.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571412426Z namespaces/openshift-dns/pods/dns-default-5dwz7/dns/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571426357Z namespaces/openshift-dns/pods/dns-default-5dwz7/dns/dns/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571438838Z namespaces/openshift-dns/pods/dns-default-5dwz7/dns/dns/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571485111Z namespaces/openshift-dns/pods/dns-default-5dwz7/dns/dns/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571570977Z namespaces/openshift-dns/pods/dns-default-5dwz7/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571636921Z namespaces/openshift-dns/pods/dns-default-5dwz7/dns/dns/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571701055Z namespaces/openshift-dns/pods/dns-default-5dwz7/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571711076Z namespaces/openshift-dns/pods/dns-default-5dwz7/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571716286Z namespaces/openshift-dns/pods/dns-default-5dwz7/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571751838Z namespaces/openshift-dns/pods/dns-default-5dwz7/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571852995Z namespaces/openshift-dns/pods/dns-default-5dwz7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571920789Z namespaces/openshift-dns/pods/dns-default-5dwz7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.571969172Z namespaces/openshift-dns/pods/dns-default-vtkgg/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572019606Z namespaces/openshift-dns/pods/dns-default-vtkgg/dns-default-vtkgg.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572124613Z namespaces/openshift-dns/pods/dns-default-vtkgg/dns/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572134593Z namespaces/openshift-dns/pods/dns-default-vtkgg/dns/dns/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572147034Z namespaces/openshift-dns/pods/dns-default-vtkgg/dns/dns/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572195597Z namespaces/openshift-dns/pods/dns-default-vtkgg/dns/dns/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572286763Z namespaces/openshift-dns/pods/dns-default-vtkgg/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572354597Z namespaces/openshift-dns/pods/dns-default-vtkgg/dns/dns/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572373009Z namespaces/openshift-dns/pods/dns-default-vtkgg/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57238325Z namespaces/openshift-dns/pods/dns-default-vtkgg/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572433753Z namespaces/openshift-dns/pods/dns-default-vtkgg/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572491266Z namespaces/openshift-dns/pods/dns-default-vtkgg/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572576702Z namespaces/openshift-dns/pods/dns-default-vtkgg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572650937Z namespaces/openshift-dns/pods/dns-default-vtkgg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572682279Z namespaces/openshift-dns/pods/node-resolver-7w5mz/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572712051Z namespaces/openshift-dns/pods/node-resolver-7w5mz/node-resolver-7w5mz.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572809547Z namespaces/openshift-dns/pods/node-resolver-7w5mz/dns-node-resolver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572817228Z namespaces/openshift-dns/pods/node-resolver-7w5mz/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572829679Z namespaces/openshift-dns/pods/node-resolver-7w5mz/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572871451Z namespaces/openshift-dns/pods/node-resolver-7w5mz/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.572958617Z namespaces/openshift-dns/pods/node-resolver-7w5mz/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573031882Z namespaces/openshift-dns/pods/node-resolver-7w5mz/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573080335Z namespaces/openshift-dns/pods/node-resolver-lw9wb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573124818Z namespaces/openshift-dns/pods/node-resolver-lw9wb/node-resolver-lw9wb.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573214074Z namespaces/openshift-dns/pods/node-resolver-lw9wb/dns-node-resolver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573226725Z namespaces/openshift-dns/pods/node-resolver-lw9wb/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573232435Z namespaces/openshift-dns/pods/node-resolver-lw9wb/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573263277Z namespaces/openshift-dns/pods/node-resolver-lw9wb/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573351353Z namespaces/openshift-dns/pods/node-resolver-lw9wb/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573473021Z namespaces/openshift-dns/pods/node-resolver-lw9wb/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573494552Z namespaces/openshift-dns/pods/node-resolver-zw2xf/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573538225Z namespaces/openshift-dns/pods/node-resolver-zw2xf/node-resolver-zw2xf.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57361371Z namespaces/openshift-dns/pods/node-resolver-zw2xf/dns-node-resolver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57362306Z namespaces/openshift-dns/pods/node-resolver-zw2xf/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573627661Z namespaces/openshift-dns/pods/node-resolver-zw2xf/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573660973Z namespaces/openshift-dns/pods/node-resolver-zw2xf/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573754919Z namespaces/openshift-dns/pods/node-resolver-zw2xf/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573821543Z namespaces/openshift-dns/pods/node-resolver-zw2xf/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573841114Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573889718Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573940391Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.573972073Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574061249Z namespaces/openshift-etcd/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574102582Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574138094Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574198588Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574257542Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574296824Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57438057Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574477856Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574570712Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574604954Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574647227Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57468576Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574741923Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574834059Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574893323Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.574926095Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.575012971Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.575096716Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.575115837Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.575253746Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.575330291Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.575414217Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.575508323Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.575588118Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.575708236Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.575864076Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57591717Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.575986204Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576010206Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576055099Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576103372Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576142724Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57622434Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576298554Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576362669Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576423242Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576487187Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576510408Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576555521Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576568112Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576621155Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576769455Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.576955877Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.577094396Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.577292879Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.577528565Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.577564127Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.577630191Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.577652623Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57776746Z namespaces/openshift-host-network/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.577800442Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.577869317Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.577906109Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.577982664Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578002745Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578100182Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578186877Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578266093Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578317186Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578366409Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578552071Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578575613Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578671189Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578724183Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578768585Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578853211Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578929516Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.578971319Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579073125Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579157481Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579247226Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579325892Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579408277Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579568367Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579669834Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579717757Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.57976198Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579829904Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579877158Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579931921Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.579976554Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.580078731Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.580136104Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.58022001Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.580302885Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.580359939Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.580453705Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.580503348Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.580546611Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.580596234Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.580609605Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.580670919Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.580942357Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.581121389Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.581265208Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.581480452Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.581689336Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.581730468Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.581778471Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.581815244Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.581897839Z namespaces/openshift-image-registry/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.581939572Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582002936Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582049669Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582094812Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582130594Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582242232Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.58237883Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582608505Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582630927Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582690521Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582737094Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582757005Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582854552Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582931866Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.582974749Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.583061695Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.583107128Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.583117158Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.58313759Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.583226216Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.583274059Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.58421517Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.584320427Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.58468168Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.584758176Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.585000811Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.58512966Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.585318892Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.585378986Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.585464722Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.585534966Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.58560069Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.585662915Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.585695926Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.585815044Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.58590285Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.585967534Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.586011887Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.586078602Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.586130835Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.586237992Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.586330058Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.586406653Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.58666434Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.586717503Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.586766016Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.586814479Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.586829651Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.586887614Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.587053855Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.587258248Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.587413849Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.587631743Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.587824115Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.587837226Z namespaces/openshift-image-registry/pods/image-registry-ff66d4bb5-czxjw/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.587860428Z namespaces/openshift-image-registry/pods/image-registry-ff66d4bb5-czxjw/image-registry-ff66d4bb5-czxjw.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.587969125Z namespaces/openshift-image-registry/pods/image-registry-ff66d4bb5-czxjw/registry/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.587978685Z namespaces/openshift-image-registry/pods/image-registry-ff66d4bb5-czxjw/registry/registry/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.587990706Z namespaces/openshift-image-registry/pods/image-registry-ff66d4bb5-czxjw/registry/registry/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.588039109Z namespaces/openshift-image-registry/pods/image-registry-ff66d4bb5-czxjw/registry/registry/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.589725039Z namespaces/openshift-image-registry/pods/image-registry-ff66d4bb5-czxjw/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.589795104Z namespaces/openshift-image-registry/pods/image-registry-ff66d4bb5-czxjw/registry/registry/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.589815985Z namespaces/openshift-image-registry/pods/node-ca-cmsm5/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.589874079Z namespaces/openshift-image-registry/pods/node-ca-cmsm5/node-ca-cmsm5.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.589948634Z namespaces/openshift-image-registry/pods/node-ca-cmsm5/node-ca/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.589959614Z namespaces/openshift-image-registry/pods/node-ca-cmsm5/node-ca/node-ca/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.589964995Z namespaces/openshift-image-registry/pods/node-ca-cmsm5/node-ca/node-ca/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590005597Z namespaces/openshift-image-registry/pods/node-ca-cmsm5/node-ca/node-ca/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590171578Z namespaces/openshift-image-registry/pods/node-ca-cmsm5/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590238382Z namespaces/openshift-image-registry/pods/node-ca-cmsm5/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590253883Z namespaces/openshift-image-registry/pods/node-ca-lfsvs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590310857Z namespaces/openshift-image-registry/pods/node-ca-lfsvs/node-ca-lfsvs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590383722Z namespaces/openshift-image-registry/pods/node-ca-lfsvs/node-ca/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590413854Z namespaces/openshift-image-registry/pods/node-ca-lfsvs/node-ca/node-ca/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590431305Z namespaces/openshift-image-registry/pods/node-ca-lfsvs/node-ca/node-ca/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590470937Z namespaces/openshift-image-registry/pods/node-ca-lfsvs/node-ca/node-ca/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590641279Z namespaces/openshift-image-registry/pods/node-ca-lfsvs/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590703163Z namespaces/openshift-image-registry/pods/node-ca-lfsvs/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590721584Z namespaces/openshift-image-registry/pods/node-ca-rv7rf/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590774387Z namespaces/openshift-image-registry/pods/node-ca-rv7rf/node-ca-rv7rf.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590848832Z namespaces/openshift-image-registry/pods/node-ca-rv7rf/node-ca/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590857923Z namespaces/openshift-image-registry/pods/node-ca-rv7rf/node-ca/node-ca/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590865933Z namespaces/openshift-image-registry/pods/node-ca-rv7rf/node-ca/node-ca/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.590917717Z namespaces/openshift-image-registry/pods/node-ca-rv7rf/node-ca/node-ca/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591099889Z namespaces/openshift-image-registry/pods/node-ca-rv7rf/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591178474Z namespaces/openshift-image-registry/pods/node-ca-rv7rf/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591220817Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591263429Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591335444Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591381977Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591459582Z namespaces/openshift-infra/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591474003Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591483714Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591531157Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591694047Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.591903061Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.592067512Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.592284446Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.592522721Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.592571544Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.592643719Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.592676861Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.592747136Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.592776898Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.592890175Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.59296702Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593064237Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593127771Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593181914Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593260279Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593310893Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593438761Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593502455Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593549338Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593653455Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593751041Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593788324Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.593916072Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.594013338Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.594160008Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.594236243Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.594412964Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.594559864Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.594741126Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.594846183Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.594895416Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.59495342Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.594973131Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.595059436Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.595080038Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.595183515Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.59526752Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.595307023Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.595355096Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.59541526Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.595546078Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.595603342Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.595618483Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.595673776Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.595838467Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.596048381Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.596204921Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.596447677Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.596767858Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.596783789Z namespaces/openshift-ingress-canary/pods/ingress-canary-dr8gm/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.596830972Z namespaces/openshift-ingress-canary/pods/ingress-canary-dr8gm/ingress-canary-dr8gm.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597058046Z namespaces/openshift-ingress-canary/pods/ingress-canary-dr8gm/serve-healthcheck-canary/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597069367Z namespaces/openshift-ingress-canary/pods/ingress-canary-dr8gm/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597078738Z namespaces/openshift-ingress-canary/pods/ingress-canary-dr8gm/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597126671Z namespaces/openshift-ingress-canary/pods/ingress-canary-dr8gm/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597246249Z namespaces/openshift-ingress-canary/pods/ingress-canary-dr8gm/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597309963Z namespaces/openshift-ingress-canary/pods/ingress-canary-dr8gm/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597361626Z namespaces/openshift-ingress-canary/pods/ingress-canary-fc7ll/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597430991Z namespaces/openshift-ingress-canary/pods/ingress-canary-fc7ll/ingress-canary-fc7ll.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597521687Z namespaces/openshift-ingress-canary/pods/ingress-canary-fc7ll/serve-healthcheck-canary/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597531127Z namespaces/openshift-ingress-canary/pods/ingress-canary-fc7ll/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597541988Z namespaces/openshift-ingress-canary/pods/ingress-canary-fc7ll/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597578911Z namespaces/openshift-ingress-canary/pods/ingress-canary-fc7ll/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597678967Z namespaces/openshift-ingress-canary/pods/ingress-canary-fc7ll/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597751702Z namespaces/openshift-ingress-canary/pods/ingress-canary-fc7ll/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597943184Z namespaces/openshift-ingress-canary/pods/ingress-canary-p2d6d/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.597987297Z namespaces/openshift-ingress-canary/pods/ingress-canary-p2d6d/ingress-canary-p2d6d.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598062382Z namespaces/openshift-ingress-canary/pods/ingress-canary-p2d6d/serve-healthcheck-canary/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598071733Z namespaces/openshift-ingress-canary/pods/ingress-canary-p2d6d/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598079113Z namespaces/openshift-ingress-canary/pods/ingress-canary-p2d6d/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598130036Z namespaces/openshift-ingress-canary/pods/ingress-canary-p2d6d/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598226693Z namespaces/openshift-ingress-canary/pods/ingress-canary-p2d6d/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598294557Z namespaces/openshift-ingress-canary/pods/ingress-canary-p2d6d/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598322699Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598355351Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598441557Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598469649Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598600867Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598632289Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598715595Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598764798Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.5988028Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.598871065Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.59894702Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.599022445Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.59910403Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.599143133Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.599204946Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.59926436Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.599307983Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.599389518Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.599453373Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.599486345Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.599575401Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.599645955Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.599689848Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.600275676Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.600357772Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60048158Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.600560515Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60063571Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.600757187Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.600967111Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601025815Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60109943Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601178175Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601224548Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601285732Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601293782Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601339285Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60141259Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601500886Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601587212Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601672367Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601736971Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601786685Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601841728Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601883461Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.601965646Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.602007819Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.602066383Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.602112336Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60217431Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.602182591Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.602219773Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.602300638Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.602308909Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.602360252Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.602611448Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60278969Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.602929189Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.603128552Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60341314Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.603474055Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.603535328Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.603580842Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.603685168Z namespaces/openshift-ingress/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60371201Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.603927324Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.603972867Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.604030161Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.604079544Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.604160379Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.604387704Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.604610058Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.604658992Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.604702784Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.604760538Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.604794741Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.604876686Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60494346Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.604978853Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.605059208Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.605114661Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.605122492Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.605168935Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.605260011Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.605342186Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.605455164Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.605499647Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.605935705Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.606039242Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.606252636Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.606330021Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.606618699Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.606735397Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60694501Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607041377Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607074989Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607173095Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607183916Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607194237Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607283713Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607317235Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607385449Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607453054Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607537439Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607618134Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607670508Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607703Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607772284Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607812407Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607881282Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607931875Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.607981278Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.608024891Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.608072574Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.608087325Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.608144039Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.608357583Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60862437Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.608794271Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.608988694Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.609198587Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.609206898Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.60924508Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/istiod-openshift-gateway-94bb8fbfd-qnh6j.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.609361938Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.609370048Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.609374049Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.609437293Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.618287369Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.618363084Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-qnh6j/discovery/discovery/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.618412317Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.618595419Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.61892027Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.618930211Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.618934682Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.618970224Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.620836735Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-mfr9j/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.620942112Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.620985725Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router-default-568468f9f4-9h48n.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621126484Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621142095Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621149576Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621174107Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621298385Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621413423Z namespaces/openshift-ingress/pods/router-default-568468f9f4-9h48n/router/router/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621466926Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.62152149Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621590824Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621625057Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621748055Z namespaces/openshift-insights/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621811229Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621891614Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.621934057Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622021943Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622076596Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622210705Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622326992Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.62244769Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622501344Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622555757Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622627932Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622657014Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622723638Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622768621Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622850697Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.62290699Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.622959794Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.623040649Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.623123194Z namespaces/openshift-insights/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.623151606Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.624099138Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.624187884Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.624658334Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.624780842Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625077402Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625165857Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625304446Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625371351Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625433855Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625503289Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625548672Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625646989Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625699642Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625747795Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.62581315Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625855262Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625915206Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.625949608Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626027504Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626065546Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626153142Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626232837Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626293191Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626327993Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626421849Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626547577Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626621492Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626654784Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626726499Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626767202Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626819075Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.626832786Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.62689423Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.627113614Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.627315487Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.627482008Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.627678821Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.627921747Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.627935518Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-dwrfw/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.62796219Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-dwrfw/insights-operator-68d9dd94cf-dwrfw.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.628065596Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-dwrfw/insights-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.628075817Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-dwrfw/insights-operator/insights-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.628083887Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-dwrfw/insights-operator/insights-operator/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.628132121Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-dwrfw/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.628561389Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-dwrfw/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.628873609Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.628908201Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/insights-runtime-extractor-5czbw.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629010178Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629020258Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/exporter/exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629025289Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/exporter/exporter/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629036419Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/exporter/exporter/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629145407Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629213471Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/exporter/exporter/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629234732Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/extractor/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629240723Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/extractor/extractor/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629252243Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/extractor/extractor/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629300826Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/extractor/extractor/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629415254Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629493239Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/extractor/extractor/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.6295139Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629547913Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629556723Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629608957Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629697872Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629763607Z namespaces/openshift-insights/pods/insights-runtime-extractor-5czbw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.62981127Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629850422Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/insights-runtime-extractor-dfl4p.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629938648Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.629950389Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/exporter/exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.62996061Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/exporter/exporter/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630003252Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/exporter/exporter/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630093328Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630163763Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/exporter/exporter/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630200355Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/extractor/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630211526Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/extractor/extractor/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630215476Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/extractor/extractor/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630261919Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/extractor/extractor/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630362166Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630455032Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/extractor/extractor/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630496304Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630639214Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630648674Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630686187Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630778573Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630848247Z namespaces/openshift-insights/pods/insights-runtime-extractor-dfl4p/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63089209Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.630942093Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/insights-runtime-extractor-q9qkd.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631025699Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63103431Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/exporter/exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63104145Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/exporter/exporter/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631087073Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/exporter/exporter/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631176359Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631243393Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/exporter/exporter/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631270055Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/extractor/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631278275Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/extractor/extractor/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631318708Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/extractor/extractor/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63134971Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/extractor/extractor/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631475148Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631543403Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/extractor/extractor/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631562664Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631576325Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631596866Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631644799Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631742336Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631802589Z namespaces/openshift-insights/pods/insights-runtime-extractor-q9qkd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631850853Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631898396Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63195599Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.631993532Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632046345Z namespaces/openshift-keda/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632055766Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632063246Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63211196Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632204056Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632251479Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632259659Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632303502Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632344995Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632379437Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632496665Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63257881Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632647625Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632660476Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632700598Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.632848078Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63302988Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.633224802Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.633443727Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.633591206Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.633768458Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63380187Z namespaces/openshift-keda/operators.coreos.com/installplans/install-kd8k7.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.633930838Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.633973821Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634028085Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634061667Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634135772Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634154543Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634284391Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634324384Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634388688Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634462443Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634536558Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.6345638Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634655165Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.6347342Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634819746Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634864809Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634902391Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.634959525Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635004258Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635084203Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635130866Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635175649Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635251074Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635311198Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635361291Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63548946Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635566675Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635655891Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635740426Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635816981Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.635936029Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636206816Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636242269Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636295522Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636344925Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636389828Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636450112Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636491575Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63657089Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636651675Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636701849Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636741741Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636799345Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636838097Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636884391Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636892681Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.636938154Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.637085464Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.637265956Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.637418415Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.637617148Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.637807261Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.637836113Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.637892806Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.637927619Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638001413Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638050287Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63811008Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638151513Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638218027Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638230908Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638313954Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63841551Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638505026Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638557069Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638601412Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638650295Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638694538Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638777894Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638824717Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638854859Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638940474Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.638993108Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.639033311Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.639133897Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.639213482Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.639293188Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.639376863Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.63947826Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.639594647Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.639768688Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.639831572Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.639876486Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.639932829Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.639979082Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640035166Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640080219Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640158464Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640237759Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640311634Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640344536Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640420041Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640475194Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640589762Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640680398Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640731441Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640807396Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640857639Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640925294Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640937455Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.640981167Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.641156869Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.641369233Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.641549924Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.641778499Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.641975822Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642029946Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642065988Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642105731Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642181156Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642215818Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642285202Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642314084Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642375708Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642515697Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642603193Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642689119Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642765703Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642822917Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642839879Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642901062Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.642947575Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.64302482Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.643074214Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.643120247Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.643200282Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.643253935Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.643293958Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.643422046Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.643503502Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.643585737Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.643665722Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.643739467Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.643868275Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644043917Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644105631Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644151214Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644207197Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.64424546Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644301444Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644348187Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644449803Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644527998Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644580132Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644612174Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644680848Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644728011Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644783525Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644805607Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644871051Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644879601Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.644906153Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.645066553Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.645247655Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.645384674Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.645596468Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.64577988Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.645809892Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.645873556Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.645916269Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.645987604Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646023316Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.64608821Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646123752Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646162885Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646217379Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646293343Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646377319Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646473555Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646532339Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646586192Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646617314Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646675238Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646755454Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646806637Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646838929Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646909564Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646973648Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.646982038Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.64701738Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.647085205Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.647121777Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.647226094Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.647303499Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.647601589Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.647680094Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.647759159Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.647882747Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648024806Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648073859Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648122783Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648158425Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648200588Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648263702Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648315455Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.64838839Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648492607Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.64854343Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648582513Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648627976Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648674599Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648720391Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648730232Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648764144Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.648923305Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.649219444Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.649355453Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.649569297Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.649750998Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.649788281Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.649840614Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.649880587Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.649942961Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.649987744Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650048898Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65008053Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650141084Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650160245Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650261182Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650333856Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650520419Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650580052Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650599054Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650657708Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65069509Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650778185Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650811098Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650860371Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.650945736Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65099945Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.651043013Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.651149Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.651234405Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65131566Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.651413107Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.651493612Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.6516184Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65177183Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.651818153Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.651857216Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.651912129Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.651957002Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652014366Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652041288Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652131544Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652208579Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652254602Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652296955Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652325786Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65237669Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652449844Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652469626Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652500848Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652654408Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652827419Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.652964898Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65315331Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653347193Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653386695Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653474021Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653514664Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653593289Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653635471Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653690015Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653730498Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653778171Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653819694Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653901929Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.653978034Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654062399Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654104402Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654146795Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654209219Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654261562Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654335287Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65438286Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654442534Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654543431Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654608845Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654654238Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654750234Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65483218Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.654906964Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65498611Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655068055Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655184152Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655340773Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655408637Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655463951Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655519574Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655558087Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655623571Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655667634Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655751349Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655830945Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655872667Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65591524Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.655963123Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.656004436Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.656051379Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.6560624Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.656106023Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.656267143Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65652891Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65667596Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.656869362Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.657066895Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.657095707Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.657155451Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.657215055Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.657335503Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.657416698Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.657504934Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.657534345Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.657611861Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.657660914Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.657772881Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65791329Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658037258Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658098512Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658144845Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65821689Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658269464Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658378961Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658536731Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658584384Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.6586844Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658728203Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658736384Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658780417Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658847611Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658893604Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.658998291Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.659085077Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.659261048Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.659338073Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.659504954Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.659626682Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.659798713Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.659859367Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.65990424Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.659958963Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660004626Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660047609Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660090042Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660201199Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660289175Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660347709Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660414303Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660498518Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660542091Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660589884Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660598175Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660656949Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.660795188Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.6609859Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66113885Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.661338623Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.661557547Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.661569088Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-8q957/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.661612761Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-8q957/kube-storage-version-migrator-operator-799cb7f958-8q957.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.661733589Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-8q957/kube-storage-version-migrator-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66174394Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-8q957/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66175294Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-8q957/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.661807604Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-8q957/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.661922321Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-8q957/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662038759Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662077172Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662132015Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662189879Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662259413Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662309256Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662409053Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662462506Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662541792Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662575504Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66267255Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662788538Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662889134Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662936888Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.662984271Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663032694Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663070316Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663154102Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663210475Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663260578Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663338203Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663385157Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663458421Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663559728Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663640423Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663759871Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663838026Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.663953043Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664069611Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664227852Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664280855Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664326588Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664373771Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664430575Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664492009Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664535521Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664618157Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664695732Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664743845Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664784288Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664833741Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664882794Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664932687Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664941438Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.664985861Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.665134981Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.665316672Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.665501275Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.665694567Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.665878639Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66589097Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.665952884Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/migrator-7f59b54f6c-klqmp.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666021748Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/graceful-termination/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666030599Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/graceful-termination/graceful-termination/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66604699Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666101003Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666213591Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666283355Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666304357Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/migrator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666343679Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/migrator/migrator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66635214Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/migrator/migrator/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66635755Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/migrator/migrator/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66651034Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666594686Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-klqmp/migrator/migrator/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666614687Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66666757Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666688582Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666752916Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666789638Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666837671Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666890445Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666935988Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.666978331Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667015843Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667105859Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667246328Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667367816Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667440711Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667503585Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667524316Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66757924Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667659255Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667697887Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.6677409Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667825176Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667867478Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667876429Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.667910721Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.668000027Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.668059991Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.668112494Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.668213991Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.668311387Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.66866061Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.668742686Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.668935688Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669063947Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669246248Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669309443Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669355015Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669443381Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669489634Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669551278Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669593031Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669679456Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669756141Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669805265Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669848478Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669908111Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.669952444Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.670005308Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.670050981Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.670096754Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.670105934Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.670149207Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.670299177Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.670526312Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.670667981Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.670889315Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671069517Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671248189Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671288331Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-6jdx7.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671375447Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671469863Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671525457Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67156999Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671622913Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671664296Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671719249Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67173375Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-7458b/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671784264Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-7458b/lws-controller-manager-558b5fdc8c-7458b.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67187361Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-7458b/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67188649Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-7458b/manager/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671891561Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-7458b/manager/manager/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.671924363Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-7458b/manager/manager/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67203959Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-7458b/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672104994Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-7458b/manager/manager/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672152998Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-pv4cf/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67219106Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-pv4cf/lws-controller-manager-558b5fdc8c-pv4cf.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672282236Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-pv4cf/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672294137Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-pv4cf/manager/manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672298927Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-pv4cf/manager/manager/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672332609Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-pv4cf/manager/manager/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672460438Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-pv4cf/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672532673Z namespaces/openshift-lws-operator/pods/lws-controller-manager-558b5fdc8c-pv4cf/manager/manager/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672580816Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dl9mp/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672625108Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dl9mp/openshift-lws-operator-fd8ccff4c-dl9mp.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672707694Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dl9mp/openshift-lws-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672716705Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dl9mp/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672727485Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dl9mp/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672764988Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dl9mp/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672925908Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dl9mp/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.672994372Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-dl9mp/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.673027645Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.673067007Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.673120661Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.673158773Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.673207486Z namespaces/openshift-machine-api/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.673217777Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.673221877Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.673242929Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67341509Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.673649895Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.673840647Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.674087254Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.674302478Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.674313578Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.674320299Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.674327409Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.674505581Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.674599227Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.674608928Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.674639039Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67479738Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.674976141Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67511185Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.675315583Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.675536298Z namespaces/openshift-marketplace/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.675554429Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67555901Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.675591501Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.675655996Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.675671387Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.675701449Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.675792535Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.675881091Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.675964866Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.676014109Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.676044231Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.676206302Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.676384253Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.676548334Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.676746547Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.676955141Z namespaces/openshift-monitoring/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.676994873Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.677062828Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67709579Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.677166624Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.677202526Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.677342346Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.677801156Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.678156849Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.678383153Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.678443587Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.678504041Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.678551504Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.678650731Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.678739927Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.678776349Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.678869885Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.678907017Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.67894227Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.682031681Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.6821743Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.683376149Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.683472235Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.684378814Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.684519943Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.684982123Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685147684Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685182046Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685339727Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685373299Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685511378Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685531069Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685635076Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685711321Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685765094Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685803287Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685923835Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.685963697Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.686052513Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.686085425Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.686207923Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.686250766Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.686354063Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.686552586Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.686640001Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.686903478Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.687036347Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.687164045Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.687255321Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.687361938Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.687480136Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.68754789Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.687598294Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.687675559Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.687758514Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.68784355Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.687956487Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688042272Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688191642Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688279318Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688374774Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688485661Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688566467Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688654563Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688729997Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688759949Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688820723Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688830604Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.688870066Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.689067439Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.689311005Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.689495407Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.68969433Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.689927265Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.689967558Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690054344Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690066944Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690130609Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690550886Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690572187Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690577288Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690585948Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690696945Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690769Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690799612Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690813833Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690822954Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690850026Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.690951332Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691018597Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691061719Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69107008Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69107414Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691107552Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691198988Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691270333Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691294254Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691301315Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691305325Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691358139Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691459345Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69153693Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691584373Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691597114Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691602535Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691629966Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691720402Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691794937Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691815278Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691823839Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69183285Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691877922Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.691963998Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692031052Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692070245Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692082546Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692089296Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692119258Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692208074Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692276528Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692339232Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-p872b/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692376715Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-p872b/cluster-monitoring-operator-5f7f494dd4-p872b.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692484722Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-p872b/cluster-monitoring-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692497893Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-p872b/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692502313Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-p872b/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.692539345Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-p872b/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.693257242Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-p872b/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.693328967Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-p872b/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69337459Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.693421273Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-state-metrics-785778ff74-8cvcc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69353603Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-main/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.693545871Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.693549881Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.693593944Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69368248Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.693747504Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.693781666Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-self/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.693788017Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.693792107Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69383272Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.693926206Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69399353Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694025142Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-state-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694037733Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694049704Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694095297Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694290479Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694354864Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-8cvcc/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694371825Z namespaces/openshift-monitoring/pods/metrics-server-74dc84475b-hznm2/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694457981Z namespaces/openshift-monitoring/pods/metrics-server-74dc84475b-hznm2/metrics-server-74dc84475b-hznm2.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694562727Z namespaces/openshift-monitoring/pods/metrics-server-74dc84475b-hznm2/metrics-server/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694575018Z namespaces/openshift-monitoring/pods/metrics-server-74dc84475b-hznm2/metrics-server/metrics-server/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694580289Z namespaces/openshift-monitoring/pods/metrics-server-74dc84475b-hznm2/metrics-server/metrics-server/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.6946084Z namespaces/openshift-monitoring/pods/metrics-server-74dc84475b-hznm2/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694701056Z namespaces/openshift-monitoring/pods/metrics-server-74dc84475b-hznm2/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694768261Z namespaces/openshift-monitoring/pods/metrics-server-74dc84475b-hznm2/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694801813Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-zpmld/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694847216Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-zpmld/monitoring-plugin-5f68df7f58-zpmld.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694925561Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-zpmld/monitoring-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694935752Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-zpmld/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694940712Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-zpmld/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.694979114Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-zpmld/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69507297Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-zpmld/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695142865Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-zpmld/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695197049Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69521519Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/node-exporter-8gvqf.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695327817Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/init-textfile/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695336658Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/init-textfile/init-textfile/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695343548Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/init-textfile/init-textfile/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695384941Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695487408Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695555852Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695601495Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695609725Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695620646Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695671129Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695761705Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695821649Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695871822Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/node-exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695880163Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/node-exporter/node-exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695887203Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/node-exporter/node-exporter/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.695933206Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696052764Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696121539Z namespaces/openshift-monitoring/pods/node-exporter-8gvqf/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696174532Z namespaces/openshift-monitoring/pods/node-exporter-chbml/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696221085Z namespaces/openshift-monitoring/pods/node-exporter-chbml/node-exporter-chbml.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696332252Z namespaces/openshift-monitoring/pods/node-exporter-chbml/init-textfile/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696349764Z namespaces/openshift-monitoring/pods/node-exporter-chbml/init-textfile/init-textfile/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696356514Z namespaces/openshift-monitoring/pods/node-exporter-chbml/init-textfile/init-textfile/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696375745Z namespaces/openshift-monitoring/pods/node-exporter-chbml/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696476212Z namespaces/openshift-monitoring/pods/node-exporter-chbml/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696543166Z namespaces/openshift-monitoring/pods/node-exporter-chbml/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696582749Z namespaces/openshift-monitoring/pods/node-exporter-chbml/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69660412Z namespaces/openshift-monitoring/pods/node-exporter-chbml/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69660817Z namespaces/openshift-monitoring/pods/node-exporter-chbml/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696614911Z namespaces/openshift-monitoring/pods/node-exporter-chbml/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696711527Z namespaces/openshift-monitoring/pods/node-exporter-chbml/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696782732Z namespaces/openshift-monitoring/pods/node-exporter-chbml/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696820714Z namespaces/openshift-monitoring/pods/node-exporter-chbml/node-exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696829705Z namespaces/openshift-monitoring/pods/node-exporter-chbml/node-exporter/node-exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696833845Z namespaces/openshift-monitoring/pods/node-exporter-chbml/node-exporter/node-exporter/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696872938Z namespaces/openshift-monitoring/pods/node-exporter-chbml/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.696974214Z namespaces/openshift-monitoring/pods/node-exporter-chbml/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697052859Z namespaces/openshift-monitoring/pods/node-exporter-chbml/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697094032Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697110293Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/node-exporter-cwh9g.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697222801Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/init-textfile/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697232241Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/init-textfile/init-textfile/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697237291Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/init-textfile/init-textfile/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697263923Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697339588Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697425294Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697455746Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697463046Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697472287Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69751689Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697608196Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69767679Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697718913Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/node-exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697728363Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/node-exporter/node-exporter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697733134Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/node-exporter/node-exporter/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697767666Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697871403Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.697938067Z namespaces/openshift-monitoring/pods/node-exporter-cwh9g/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69797667Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698020062Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/openshift-state-metrics-84bd68b896-kbbbg.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698151331Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-main/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698163232Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698169172Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698208945Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698300901Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698367375Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698412828Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-self/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698422349Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698432329Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698495273Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698583409Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698652244Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698721278Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/openshift-state-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698731979Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698739389Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69874564Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698857757Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698925881Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-kbbbg/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.698990336Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69905929Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699243702Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699253493Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699259963Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699299346Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699416793Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699502969Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699543342Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699553282Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699560613Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699598795Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699682471Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699751755Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699777007Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699785617Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699789658Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.69982715Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699920906Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.699988901Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700025113Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700036974Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700042734Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700073906Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700152351Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700220606Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700255498Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700264168Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700269979Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700307681Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700414208Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700482723Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700529216Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700544467Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700549857Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.700564588Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701127575Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701194699Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701235382Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701248353Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701254223Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701277215Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701375831Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701468597Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701499709Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701561413Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/prometheus-operator-64785b6675-z55bt.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701680101Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701690272Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701694392Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701721634Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701811339Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701882584Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701920556Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/prometheus-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701929267Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.701936048Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.70198069Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702224766Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702288671Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-z55bt/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702332933Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-87596/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702373386Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-87596/prometheus-operator-admission-webhook-77d98bc786-87596.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702512745Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-87596/prometheus-operator-admission-webhook/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702528866Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-87596/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702534756Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-87596/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702549357Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-87596/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702659015Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-87596/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702731319Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-87596/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702751121Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702800324Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/telemeter-client-5b9c457dbb-5zf5d.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702945103Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702955614Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702959754Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.702986916Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703079102Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703144186Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703183789Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/reload/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703192129Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/reload/reload/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703196379Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/reload/reload/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703235732Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/reload/reload/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703330498Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703414604Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/reload/reload/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703491299Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/telemeter-client/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703500069Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/telemeter-client/telemeter-client/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703504139Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703537152Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703677551Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703747206Z namespaces/openshift-monitoring/pods/telemeter-client-5b9c457dbb-5zf5d/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703780498Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703846812Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/thanos-querier-6769966f79-57q7b.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.70397116Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.70397923Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.703987881Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704025954Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704110879Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704185884Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704220736Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-rules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704226927Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704230957Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.70427096Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704356735Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704442101Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704465172Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-web/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704485474Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704494564Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704537797Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704622943Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704692417Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704708338Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704717999Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704722359Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704785603Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.704916102Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705002707Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705052711Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/prom-label-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705067012Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705071192Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705131056Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705250233Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.70534836Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705415114Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/thanos-query/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705439296Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/thanos-query/thanos-query/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705446686Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/thanos-query/thanos-query/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705527542Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705670781Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705755886Z namespaces/openshift-monitoring/pods/thanos-querier-6769966f79-57q7b/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.70581292Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705856833Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705930468Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.705974551Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7061195Z namespaces/openshift-multus/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.706174924Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.706305602Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.706348965Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.706421009Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.706477423Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.70673204Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.706823056Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.706929443Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.706990107Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.70704977Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.707239343Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.707419865Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.707514591Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.707572575Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.707631988Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.707691342Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.707799319Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.707860923Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.707911667Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.708021144Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.708100089Z namespaces/openshift-multus/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.708142622Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.708287221Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.708380517Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.708982787Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.709113095Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.709719334Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.70980118Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.709928578Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.709965571Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710021814Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71011296Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710196406Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710278271Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710370647Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710407139Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710536538Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710623073Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710703079Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710761022Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710802965Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710860509Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71087564Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710940174Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.710986377Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711040141Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711079373Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711163478Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711239734Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71133702Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711378963Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711459418Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711501571Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711546454Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711581716Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711629999Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7116486Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711695363Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.711844013Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.712026365Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.712165864Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.712364827Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.712730871Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.712747032Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.712786194Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/multus-additional-cni-plugins-6gzcp.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.712935594Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/bond-cni-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.712949375Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.712954615Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.712986397Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713075733Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713143898Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713166179Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7131793Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/cni-plugins/cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713190761Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713222883Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713317319Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713379663Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713434296Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/egress-router-binary-copy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713450277Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713457928Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71348991Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713573245Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71364988Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713668772Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713674952Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713716565Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713765548Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713840613Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713900477Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713927638Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/routeoverride-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713932439Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713937109Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.713985022Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714066627Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714136532Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714167544Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni-bincopy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714177695Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714182745Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714217117Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714304643Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714373138Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71441506Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714429091Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714446662Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714473674Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714576051Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714647785Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6gzcp/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714666126Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714728311Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/multus-additional-cni-plugins-mzfvv.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714858429Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/bond-cni-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71486638Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71487041Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.714890421Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715002319Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715051002Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715092604Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715100965Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/cni-plugins/cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715104985Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715124816Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715220653Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715289127Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71533192Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/egress-router-binary-copy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715344561Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715350601Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715372953Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71548264Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715549894Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715589107Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715594757Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715598938Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71563084Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715706834Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715772349Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71579197Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/routeoverride-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715805011Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715811101Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.715852694Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71594543Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716012284Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716049427Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni-bincopy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716055637Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716059627Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71609484Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716182185Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71624826Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716274701Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716279192Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716285802Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716345996Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716448463Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716513547Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-mzfvv/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71655649Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716602373Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/multus-additional-cni-plugins-t9r69.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716736941Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/bond-cni-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716747212Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716753762Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716803796Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716895402Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716968836Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716983848Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716989098Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/cni-plugins/cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.716994898Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717035611Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717126777Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717194161Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717234854Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/egress-router-binary-copy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717244374Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717249075Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717276387Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717376013Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717464159Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71748336Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717496891Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717501751Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717542784Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717614779Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717690463Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717705404Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/routeoverride-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717717305Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717724856Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717767219Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717850334Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717922329Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717959621Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni-bincopy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717965801Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.717971312Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718008234Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71809031Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718159884Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718178226Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718188336Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718194176Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.71824121Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718329815Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718409441Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t9r69/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718449563Z namespaces/openshift-multus/pods/multus-lsdhr/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718502707Z namespaces/openshift-multus/pods/multus-lsdhr/multus-lsdhr.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718599133Z namespaces/openshift-multus/pods/multus-lsdhr/kube-multus/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718608674Z namespaces/openshift-multus/pods/multus-lsdhr/kube-multus/kube-multus/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718617144Z namespaces/openshift-multus/pods/multus-lsdhr/kube-multus/kube-multus/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.718643756Z namespaces/openshift-multus/pods/multus-lsdhr/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.719030971Z namespaces/openshift-multus/pods/multus-lsdhr/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.719096445Z namespaces/openshift-multus/pods/multus-lsdhr/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.719125047Z namespaces/openshift-multus/pods/multus-qj6b6/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.719159479Z namespaces/openshift-multus/pods/multus-qj6b6/multus-qj6b6.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.719256616Z namespaces/openshift-multus/pods/multus-qj6b6/kube-multus/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.719265256Z namespaces/openshift-multus/pods/multus-qj6b6/kube-multus/kube-multus/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.719269246Z namespaces/openshift-multus/pods/multus-qj6b6/kube-multus/kube-multus/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.719302719Z namespaces/openshift-multus/pods/multus-qj6b6/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.719979583Z namespaces/openshift-multus/pods/multus-qj6b6/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720049687Z namespaces/openshift-multus/pods/multus-qj6b6/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720065458Z namespaces/openshift-multus/pods/multus-zl4qh/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720125422Z namespaces/openshift-multus/pods/multus-zl4qh/multus-zl4qh.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720283892Z namespaces/openshift-multus/pods/multus-zl4qh/kube-multus/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720298344Z namespaces/openshift-multus/pods/multus-zl4qh/kube-multus/kube-multus/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720306134Z namespaces/openshift-multus/pods/multus-zl4qh/kube-multus/kube-multus/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720318145Z namespaces/openshift-multus/pods/multus-zl4qh/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720685069Z namespaces/openshift-multus/pods/multus-zl4qh/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720742112Z namespaces/openshift-multus/pods/multus-zl4qh/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720760614Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720823308Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/network-metrics-daemon-27zcc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720913724Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720922324Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720926924Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.720970837Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721065713Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721130307Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72117272Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/network-metrics-daemon/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721181951Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721191442Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721228954Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721427437Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721494521Z namespaces/openshift-multus/pods/network-metrics-daemon-27zcc/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721529904Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721575307Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/network-metrics-daemon-kvr2r.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721651032Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721659472Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721664162Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721698995Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72178845Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721855095Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721884557Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/network-metrics-daemon/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721889217Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721896768Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.721945771Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722096101Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722161495Z namespaces/openshift-multus/pods/network-metrics-daemon-kvr2r/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722177876Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722232339Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/network-metrics-daemon-zqljw.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722316855Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722326766Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722331856Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722354807Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722462314Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72254397Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722562811Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/network-metrics-daemon/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722573182Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722578792Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722628105Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722759154Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722827518Z namespaces/openshift-multus/pods/network-metrics-daemon-zqljw/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722871591Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722904943Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722954006Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722965807Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.722977888Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.723063544Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.723118407Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72316526Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.723240925Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.723290678Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72332497Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.723373924Z namespaces/openshift-must-gather-qh77t/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.723385545Z namespaces/openshift-must-gather-qh77t/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.723391965Z namespaces/openshift-must-gather-qh77t/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.723437858Z namespaces/openshift-must-gather-qh77t/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7236271Z namespaces/openshift-must-gather-qh77t/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.723818983Z namespaces/openshift-must-gather-qh77t/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.723965932Z namespaces/openshift-must-gather-qh77t/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.724159395Z namespaces/openshift-must-gather-qh77t/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.724344607Z namespaces/openshift-network-console/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72439169Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.724467815Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.724501837Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.724551761Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.724597353Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.724679599Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.724793666Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.724895243Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.724939366Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.724971098Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725048673Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725090996Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725139909Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725172231Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725259777Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72530511Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725336862Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725442048Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725487851Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725523624Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725635311Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725719066Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725867556Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.725934Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.726102462Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72622676Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.726441353Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.726610375Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.726620505Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.726625696Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.726632056Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72669248Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.726740973Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.726796157Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72683965Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.726876862Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.726911214Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72699593Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.727074425Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.727119818Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.727160971Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.727195423Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.727243986Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.727286749Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.727294769Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.727335872Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.727585378Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72776709Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.727908809Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728101492Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728297264Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728308185Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-tnwtv/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728340987Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-tnwtv/networking-console-plugin-6cf856758c-tnwtv.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728440364Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-tnwtv/networking-console-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728451284Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-tnwtv/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728455505Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-tnwtv/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728494177Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-tnwtv/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728583153Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-tnwtv/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728652897Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-tnwtv/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728671469Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728720392Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728773805Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728815908Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72884707Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728904934Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.728962898Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72900001Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729047013Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729082906Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729187572Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729286259Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729379265Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729444239Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729484322Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729553356Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729600009Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729668284Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729686235Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729751749Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729792402Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729872677Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.72991799Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.729955153Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.730034778Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.730083711Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.730119843Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.73021845Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.730308985Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.73053507Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.730613495Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.730817999Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.730909024Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731006911Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731065495Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731093007Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731174262Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731219615Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.73130632Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731355784Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731417358Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731490382Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731535995Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731579968Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731620031Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731701626Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731777861Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731812493Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731862967Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.73191767Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.731955953Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.732006826Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.732046018Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.732088331Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.732096892Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.732137955Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.732285764Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.732544701Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.73268214Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.732872212Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733056304Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733065725Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-bp4xd/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733099377Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-bp4xd/network-check-source-869dcfcdcd-bp4xd.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733186913Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-bp4xd/check-endpoints/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733195144Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-bp4xd/check-endpoints/check-endpoints/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733199284Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-bp4xd/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733231906Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-bp4xd/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733326412Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-bp4xd/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733408037Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-bp4xd/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733457981Z namespaces/openshift-network-diagnostics/pods/network-check-target-5w757/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733501133Z namespaces/openshift-network-diagnostics/pods/network-check-target-5w757/network-check-target-5w757.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733577258Z namespaces/openshift-network-diagnostics/pods/network-check-target-5w757/network-check-target-container/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733587759Z namespaces/openshift-network-diagnostics/pods/network-check-target-5w757/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733594119Z namespaces/openshift-network-diagnostics/pods/network-check-target-5w757/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733622721Z namespaces/openshift-network-diagnostics/pods/network-check-target-5w757/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733711527Z namespaces/openshift-network-diagnostics/pods/network-check-target-5w757/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733783212Z namespaces/openshift-network-diagnostics/pods/network-check-target-5w757/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733820014Z namespaces/openshift-network-diagnostics/pods/network-check-target-6cn29/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733853016Z namespaces/openshift-network-diagnostics/pods/network-check-target-6cn29/network-check-target-6cn29.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733929701Z namespaces/openshift-network-diagnostics/pods/network-check-target-6cn29/network-check-target-container/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733941962Z namespaces/openshift-network-diagnostics/pods/network-check-target-6cn29/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733948322Z namespaces/openshift-network-diagnostics/pods/network-check-target-6cn29/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.733971274Z namespaces/openshift-network-diagnostics/pods/network-check-target-6cn29/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.73406006Z namespaces/openshift-network-diagnostics/pods/network-check-target-6cn29/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734124704Z namespaces/openshift-network-diagnostics/pods/network-check-target-6cn29/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734163707Z namespaces/openshift-network-diagnostics/pods/network-check-target-g4r9z/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734182828Z namespaces/openshift-network-diagnostics/pods/network-check-target-g4r9z/network-check-target-g4r9z.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734276124Z namespaces/openshift-network-diagnostics/pods/network-check-target-g4r9z/network-check-target-container/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734284684Z namespaces/openshift-network-diagnostics/pods/network-check-target-g4r9z/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734291365Z namespaces/openshift-network-diagnostics/pods/network-check-target-g4r9z/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734323017Z namespaces/openshift-network-diagnostics/pods/network-check-target-g4r9z/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734406702Z namespaces/openshift-network-diagnostics/pods/network-check-target-g4r9z/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734507589Z namespaces/openshift-network-diagnostics/pods/network-check-target-g4r9z/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734535981Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734581304Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734628457Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734637087Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.73466942Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734755725Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734807778Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734847071Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.734930957Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.73498178Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735000571Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735067275Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735114178Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735177873Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735194594Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735254267Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735301321Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735375786Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735547407Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735625292Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735677975Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735715437Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735764201Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735809564Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735889459Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735935412Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.735978025Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.73605838Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736103923Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736116514Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736149786Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736196889Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736243082Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736338408Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736445825Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736531901Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736611016Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736689341Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.73682801Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736945388Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.736995411Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737038394Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737090267Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.73713336Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737179593Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737219826Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737266839Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737308271Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737387597Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737490823Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737532796Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737573679Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737617112Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737661314Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737707647Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.737715738Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7377517Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7379008Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738108204Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738243242Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738456086Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738650829Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738693082Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738741585Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738749735Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738784597Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738840851Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738871633Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738924426Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.738969549Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739012172Z namespaces/openshift-network-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739055045Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739120489Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739161832Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739211475Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739248348Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739357085Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739468352Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739532746Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739579119Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739621412Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739759721Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739797763Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739847177Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739887099Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.739968274Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.740014958Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.74005695Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.740137716Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.740188049Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.74019937Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.740220011Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.740273534Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.740316097Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.740471858Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.740553913Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.740709673Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.740788798Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.740956969Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741043455Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741171533Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741225567Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741268369Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741352855Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741411648Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741474623Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741525866Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741560558Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741618092Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741656584Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741702538Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741751181Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741826846Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741905111Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741951924Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.741992576Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.74204514Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.742090443Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.742141716Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.742185189Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.742234872Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.742279525Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.742323048Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.742331728Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.742368321Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.742546582Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.742728474Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.742873224Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743068467Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743255249Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743267049Z namespaces/openshift-network-operator/pods/iptables-alerter-2fzlm/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743295451Z namespaces/openshift-network-operator/pods/iptables-alerter-2fzlm/iptables-alerter-2fzlm.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743373006Z namespaces/openshift-network-operator/pods/iptables-alerter-2fzlm/iptables-alerter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743386897Z namespaces/openshift-network-operator/pods/iptables-alerter-2fzlm/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743391088Z namespaces/openshift-network-operator/pods/iptables-alerter-2fzlm/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743524956Z namespaces/openshift-network-operator/pods/iptables-alerter-2fzlm/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743611092Z namespaces/openshift-network-operator/pods/iptables-alerter-2fzlm/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743680627Z namespaces/openshift-network-operator/pods/iptables-alerter-2fzlm/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743698908Z namespaces/openshift-network-operator/pods/iptables-alerter-7bfv6/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743753951Z namespaces/openshift-network-operator/pods/iptables-alerter-7bfv6/iptables-alerter-7bfv6.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743832196Z namespaces/openshift-network-operator/pods/iptables-alerter-7bfv6/iptables-alerter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743843717Z namespaces/openshift-network-operator/pods/iptables-alerter-7bfv6/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743849297Z namespaces/openshift-network-operator/pods/iptables-alerter-7bfv6/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743874099Z namespaces/openshift-network-operator/pods/iptables-alerter-7bfv6/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.743961165Z namespaces/openshift-network-operator/pods/iptables-alerter-7bfv6/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744029919Z namespaces/openshift-network-operator/pods/iptables-alerter-7bfv6/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744060221Z namespaces/openshift-network-operator/pods/iptables-alerter-9mk4h/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744100784Z namespaces/openshift-network-operator/pods/iptables-alerter-9mk4h/iptables-alerter-9mk4h.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744173159Z namespaces/openshift-network-operator/pods/iptables-alerter-9mk4h/iptables-alerter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744184989Z namespaces/openshift-network-operator/pods/iptables-alerter-9mk4h/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.74419093Z namespaces/openshift-network-operator/pods/iptables-alerter-9mk4h/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744220911Z namespaces/openshift-network-operator/pods/iptables-alerter-9mk4h/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744310537Z namespaces/openshift-network-operator/pods/iptables-alerter-9mk4h/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744375202Z namespaces/openshift-network-operator/pods/iptables-alerter-9mk4h/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744418455Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744468308Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744521721Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744557904Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744591116Z namespaces/openshift-node/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744596296Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744606217Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744647209Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.74480771Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.744998312Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.745131891Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.745414989Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.745602761Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.745641404Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.745704138Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.745745341Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.745792854Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.745834007Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.745923042Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.745995967Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746074272Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746128816Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746159228Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746214681Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746244263Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746336399Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746377232Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746439216Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746535142Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746584396Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746603787Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746717684Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.74680699Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746896576Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.746975001Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747053156Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747171504Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747353726Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747437241Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747479454Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747539038Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.74757221Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747654035Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747700248Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747812936Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747890981Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747942314Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.747973896Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.74802906Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.748061522Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.748127176Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.748155768Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.748237903Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.748247404Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.748257225Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.748446977Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.748639209Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.748770838Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.74896007Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749151333Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749168854Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749242049Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749286072Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749343845Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749372547Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749455612Z namespaces/openshift-operators/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749475104Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749481244Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749496115Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749643075Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749819826Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.749959595Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.750153878Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.750442917Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.750476179Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.750545323Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.750615068Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.750697553Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.750742086Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.75079206Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.750838163Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751028815Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.75110626Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751184655Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751230688Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751275181Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.75141906Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751537728Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751584901Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751626404Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751705459Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751751252Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751783064Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.75186893Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751912743Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751921803Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.751955105Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.752007439Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.752050041Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.752248754Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.75233303Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.752695563Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.752814791Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.753274701Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.753361267Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.753590272Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.753646576Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.753697079Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.753787005Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.753904552Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.753933364Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.75402725Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.754075093Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.754116596Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.75417036Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.754212432Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.769716842Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.769752765Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.76983682Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.769847481Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.769875542Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.769946137Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.76999452Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.770101997Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.770196323Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.770250407Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.77029837Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.770364674Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.770531495Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.770648003Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.770701866Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.770773721Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.770785952Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.770821354Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.77092149Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.770968714Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.771010046Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.771056629Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.77106483Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.771104433Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.771250892Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.771461836Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.771606495Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.771802608Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.771999301Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772009412Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772058005Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovnkube-node-7wsjz.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772230776Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-node/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772239167Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772243177Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772275339Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772418628Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772503444Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772540516Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772545727Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772554327Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772587399Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772709857Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772775671Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772810944Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/nbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772819654Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/nbdb/nbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772823645Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/nbdb/nbdb/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772858317Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/nbdb/nbdb/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.772948753Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773025058Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7730535Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/northd/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7730589Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/northd/northd/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.77306365Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/northd/northd/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773101633Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/northd/northd/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773203469Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773269314Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/northd/northd/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773315597Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovn-acl-logging/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773327447Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773332498Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773340038Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773479527Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773504729Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovn-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.77351803Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovn-controller/ovn-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7735224Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773580734Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773775736Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773853681Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773906845Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovnkube-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773915946Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773920086Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.773958188Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775222901Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775289045Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775330778Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/sbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775340449Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/sbdb/sbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775344539Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/sbdb/sbdb/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.77537208Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/sbdb/sbdb/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775488038Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775564603Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-7wsjz/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775602795Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.77566229Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovnkube-node-h7t4d.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.77582404Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-node/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775835191Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775840091Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775853872Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.775987291Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776054385Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776083587Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776088217Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776092247Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.77613535Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776256388Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776324542Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776362185Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/nbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776369965Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/nbdb/nbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776373886Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/nbdb/nbdb/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776411038Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/nbdb/nbdb/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776531196Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.77659738Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776636293Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/northd/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776645023Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/northd/northd/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776649174Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/northd/northd/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776672295Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/northd/northd/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776778002Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776848297Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/northd/northd/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776864338Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-acl-logging/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776875778Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776880049Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.776921101Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777012217Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777088712Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777104913Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777111064Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-controller/ovn-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777115444Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777158867Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777419834Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777503209Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777536761Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovnkube-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777546352Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777550542Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.777581094Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.778939793Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779009527Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779026849Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/sbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779034539Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/sbdb/sbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779038479Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/sbdb/sbdb/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779094493Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/sbdb/sbdb/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779182369Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779270345Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-h7t4d/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779296846Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779411994Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovnkube-node-xvjmz.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779580905Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-node/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779591265Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779596526Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779632788Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779801189Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779870864Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779920007Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779928317Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.779932448Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.77997288Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7801289Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780217996Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78027837Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/nbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780287201Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/nbdb/nbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780291461Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/nbdb/nbdb/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780324743Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/nbdb/nbdb/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780462112Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780542027Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78058472Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/northd/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780594211Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/northd/northd/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780598261Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/northd/northd/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780643764Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/northd/northd/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780766942Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780854388Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/northd/northd/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78088628Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovn-acl-logging/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78089089Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78089472Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.780945874Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78105279Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.781114965Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovn-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.781123205Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovn-controller/ovn-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.781127505Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.781166378Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.781418744Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78150243Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.781530952Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovnkube-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.781535642Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.781539682Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.781590085Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.782965365Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783050751Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783075132Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/sbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783082233Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/sbdb/sbdb/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783094034Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/sbdb/sbdb/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783147017Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/sbdb/sbdb/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783269215Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78335225Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-xvjmz/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783369651Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783454717Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78350432Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783513751Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783577985Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783669301Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78380441Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783850393Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783879025Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.783977331Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784063977Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78411268Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784151642Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784199095Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784211606Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784217127Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784224017Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784295302Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784303982Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784337924Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784567659Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784780403Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.784963555Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.785165508Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.785356701Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.785410584Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.785482899Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.785524412Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.785574125Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.785606707Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.785689863Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7858008Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.785896886Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.785937989Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.785985012Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786034445Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786065277Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786150473Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786197285Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786228948Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786313513Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786345565Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786350195Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78642171Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786474924Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786528317Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786649645Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78672984Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.786922323Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787002168Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787115045Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787226383Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787418845Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78749177Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787516871Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787589596Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787603737Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787669761Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787706204Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787788479Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787875615Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787907397Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7879621Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.787998793Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.788054546Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78810601Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.788124511Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.788185845Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.788195186Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.788221267Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.788375617Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78857468Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.788710199Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.788911152Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789092634Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789101605Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-hvxkc/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789126826Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-hvxkc/service-ca-operator-7695fc6c8c-hvxkc.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789248384Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-hvxkc/service-ca-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789257565Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-hvxkc/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789261735Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-hvxkc/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789298378Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-hvxkc/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789472399Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-hvxkc/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789632359Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789663621Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789722325Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789739196Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789804251Z namespaces/openshift-service-ca/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789848023Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789898627Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.78994615Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.789990383Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790031685Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790112431Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790214537Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790313764Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790358307Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7904113Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790466064Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790527428Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790600312Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790651766Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790670207Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790761903Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790808166Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790816286Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790852149Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790903592Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.790936954Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791044011Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791120476Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791243354Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791314089Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791448798Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791568175Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791731166Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791781719Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791826682Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791869705Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791916538Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.791961011Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.792001214Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.792083299Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.792160524Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.792207937Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79224937Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.792290112Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.792338265Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.792385068Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79240992Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.792497596Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.792648696Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.792962506Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.793100715Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.793295638Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.793511662Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.793524563Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-8mpmw/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.793555045Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-8mpmw/service-ca-79f68b9f4f-8mpmw.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79363321Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-8mpmw/service-ca-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.793642301Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-8mpmw/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.793646471Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-8mpmw/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.793678913Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-8mpmw/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.793894287Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-8mpmw/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.793960641Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-8mpmw/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794002814Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794033736Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794085319Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794126462Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794173685Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794219188Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794279062Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794317285Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794364677Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794412531Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794521228Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794599383Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794677068Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794729561Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794762914Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794811947Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794843409Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794936385Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.794981068Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79502206Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.795104276Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.795151719Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.795184091Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.795289008Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.795367373Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79547042Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.795549625Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79562922Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.795741447Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.795895627Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79594387Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.795993003Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796037246Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796079169Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796125212Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796164915Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.7962449Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796326075Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796372248Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796409181Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796469505Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796517208Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796565731Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796573701Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796608334Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796755923Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.796945726Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.797083295Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.797270247Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79747516Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.797512373Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.797562386Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.797593218Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.797649602Z namespaces/openshift/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.797693854Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.797748688Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79778856Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.797835394Z namespaces/openshift/apps/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.797876326Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.797957542Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798036407Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798112932Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798161005Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798202838Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79824943Z namespaces/openshift/batch/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798282993Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798361088Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798500487Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798521688Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798618114Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798665188Z namespaces/openshift/core/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79870421Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798801897Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798875291Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.798958337Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.799037122Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.799127488Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.799245055Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.799413866Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.79947238Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.799517683Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.799564066Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.799600338Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.800914984Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.800942896Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.801028011Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.801108457Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.801210783Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.80131749Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.801503553Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.801659463Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.801818173Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.801940071Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.802034447Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.802136174Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.802251041Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.802367609Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.802501807Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.802603084Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.802682169Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.802788626Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.802902444Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.803016361Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.803113467Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.803212264Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.80330482Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.803420497Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.803535265Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.803632091Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.803726247Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.803834044Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.803933671Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.804045088Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.804164556Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.804271623Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.80438205Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.804489597Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.804597914Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.80469375Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.804812958Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.804958607Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.805045553Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.805165981Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.805263527Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.805376815Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.805528515Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.805628531Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.805716037Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.805845865Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.805984024Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.806090401Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.806199818Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.806291764Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.80638487Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.806509919Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.806623666Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.806749564Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.806862611Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.806977329Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807095557Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807190613Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807287739Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807420518Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807507173Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807529885Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.80761512Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807701186Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807749349Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807786142Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807835645Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807872677Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.80792272Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807930011Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.807966374Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.808169597Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.808360079Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.808577663Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.808780186Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.808952158Z namespaces/openshift/policy/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.8089801Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.809034783Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.809078316Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.809129009Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.80913727Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.809162541Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.809274008Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.809416928Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.809555627Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.809666714Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.809829475Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.809941422Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.810047309Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.810145936Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.810274654Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.810411283Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.810544011Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.810665959Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.810803338Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.810908995Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.811026263Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.8111328Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.811244457Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.811352274Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.811481122Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.81159929Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.811709737Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.811820394Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.811920671Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.812022608Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.812126224Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.812228191Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.812330958Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.812469897Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.812580074Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.812684271Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.812783897Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.812897215Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.813019743Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.813217745Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.813321092Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.813451051Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.813553527Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.813672995Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.813782892Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.813883809Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.814005727Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.814123995Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.814244072Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.814369431Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.814537142Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.814680161Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.814805449Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.814949478Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.815059645Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.815154702Z network_logs/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.815185814Z network_logs/cluster_scale [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.815274739Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.815349024Z network_logs/multi-networkpolicy [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.815450971Z network_logs/net-attach-def [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.815541047Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.815680706Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.815772612Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.817630073Z nodes/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.817652354Z nodes/debug [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.817755981Z nodes/ip-10-0-128-42.ec2.internal/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.817788243Z nodes/ip-10-0-128-42.ec2.internal/cpu_affinities.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.818058411Z nodes/ip-10-0-128-42.ec2.internal/dmesg [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.818333969Z nodes/ip-10-0-128-42.ec2.internal/ethtool_channels [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.818407963Z nodes/ip-10-0-128-42.ec2.internal/ethtool_features [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.818536742Z nodes/ip-10-0-128-42.ec2.internal/ip-10-0-128-42.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.818913847Z nodes/ip-10-0-128-42.ec2.internal/irq_affinities.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.819056106Z nodes/ip-10-0-128-42.ec2.internal/lscpu [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.819189574Z nodes/ip-10-0-128-42.ec2.internal/lspci [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.819290071Z nodes/ip-10-0-128-42.ec2.internal/podresources.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.819391707Z nodes/ip-10-0-128-42.ec2.internal/pods_info.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.819533887Z nodes/ip-10-0-128-42.ec2.internal/proc_cmdline [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.819616042Z nodes/ip-10-0-128-42.ec2.internal/sysinfo.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.82141879Z nodes/ip-10-0-128-42.ec2.internal/sysinfo.tgz [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.82157989Z nodes/ip-10-0-132-130.ec2.internal/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.821620903Z nodes/ip-10-0-132-130.ec2.internal/cpu_affinities.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.821974156Z nodes/ip-10-0-132-130.ec2.internal/dmesg [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.822221522Z nodes/ip-10-0-132-130.ec2.internal/ethtool_channels [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.822298037Z nodes/ip-10-0-132-130.ec2.internal/ethtool_features [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.822380402Z nodes/ip-10-0-132-130.ec2.internal/ip-10-0-132-130.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.822823551Z nodes/ip-10-0-132-130.ec2.internal/irq_affinities.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.822922517Z nodes/ip-10-0-132-130.ec2.internal/lscpu [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.823044185Z nodes/ip-10-0-132-130.ec2.internal/lspci [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.823133841Z nodes/ip-10-0-132-130.ec2.internal/podresources.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.823241028Z nodes/ip-10-0-132-130.ec2.internal/pods_info.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.823366617Z nodes/ip-10-0-132-130.ec2.internal/proc_cmdline [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.823475024Z nodes/ip-10-0-132-130.ec2.internal/sysinfo.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.825318474Z nodes/ip-10-0-132-130.ec2.internal/sysinfo.tgz [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.825501236Z nodes/ip-10-0-133-160.ec2.internal/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.825527127Z nodes/ip-10-0-133-160.ec2.internal/cpu_affinities.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.825822906Z nodes/ip-10-0-133-160.ec2.internal/dmesg [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.826067202Z nodes/ip-10-0-133-160.ec2.internal/ethtool_channels [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.826144117Z nodes/ip-10-0-133-160.ec2.internal/ethtool_features [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.826227573Z nodes/ip-10-0-133-160.ec2.internal/ip-10-0-133-160.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.826591747Z nodes/ip-10-0-133-160.ec2.internal/irq_affinities.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.826722755Z nodes/ip-10-0-133-160.ec2.internal/lscpu [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.826841723Z nodes/ip-10-0-133-160.ec2.internal/lspci [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.826933349Z nodes/ip-10-0-133-160.ec2.internal/podresources.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.827035445Z nodes/ip-10-0-133-160.ec2.internal/pods_info.json [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.827174024Z nodes/ip-10-0-133-160.ec2.internal/proc_cmdline [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.82725642Z nodes/ip-10-0-133-160.ec2.internal/sysinfo.log [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.829131782Z nodes/ip-10-0-133-160.ec2.internal/sysinfo.tgz [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.829306053Z pod_network_connectivity_check/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.829325065Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.829472004Z static-pods/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.829487085Z static-pods/kube-apiserver/ [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.836766009Z [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.836791781Z sent 71,598 bytes received 7,138,409 bytes 4,806,671.33 bytes/sec [must-gather] [must-gather-82ktg] OUT 2026-07-16T18:28:55.836800361Z total size is 89,731,684 speedup is 12.45 [must-gather] [must-gather ] OUT 2026-07-16T18:28:56.131772942Z namespace/openshift-must-gather-qh77t deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: a9b59be7-cab2-4d29-b5e3-23627224b3ec [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-r77cn [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-r77cn-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-r77cn [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 14608059 Jul 16 18:30 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-r77cn/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts a2f18ff] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-r77cn/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] b19f2e8..a2f18ff ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-16T18:30:30.418Z","type":3}]