task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-nrqt5-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-nrqt5-kubeconfig [get-kubeconfig] Found admin password secret: cluster-nrqt5-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-nrqt5-password [get-kubeconfig] API Server URL: https://abc95519c67f9460189748464d737b7a-a75d8cad7bdd6449.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.0bbc98b4-c464-4493-b02e-7e9bfa164e3b.prod.konfluxeaas.com [clone-repo] codex/llmisvc-workload-tls-profile-midstream [clone-repo] https://github.com/andresllh/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 48% (1770/3619) Updating files: 49% (1774/3619) Updating files: 50% (1810/3619) Updating files: 51% (1846/3619) Updating files: 52% (1882/3619) Updating files: 53% (1919/3619) Updating files: 54% (1955/3619) Updating files: 55% (1991/3619) Updating files: 56% (2027/3619) Updating files: 57% (2063/3619) Updating files: 58% (2100/3619) Updating files: 59% (2136/3619) Updating files: 60% (2172/3619) Updating files: 61% (2208/3619) Updating files: 62% (2244/3619) Updating files: 63% (2280/3619) Updating files: 64% (2317/3619) Updating files: 65% (2353/3619) Updating files: 66% (2389/3619) Updating files: 67% (2425/3619) Updating files: 68% (2461/3619) Updating files: 69% (2498/3619) Updating files: 70% (2534/3619) Updating files: 71% (2570/3619) Updating files: 72% (2606/3619) Updating files: 73% (2642/3619) Updating files: 74% (2679/3619) Updating files: 75% (2715/3619) Updating files: 76% (2751/3619) Updating files: 77% (2787/3619) Updating files: 78% (2823/3619) Updating files: 79% (2860/3619) Updating files: 80% (2896/3619) Updating files: 81% (2932/3619) Updating files: 82% (2968/3619) Updating files: 83% (3004/3619) Updating files: 84% (3040/3619) Updating files: 85% (3077/3619) Updating files: 86% (3113/3619) Updating files: 87% (3149/3619) Updating files: 88% (3185/3619) Updating files: 89% (3221/3619) Updating files: 90% (3258/3619) Updating files: 91% (3294/3619) Updating files: 92% (3330/3619) Updating files: 93% (3366/3619) Updating files: 94% (3402/3619) Updating files: 95% (3439/3619) Updating files: 96% (3475/3619) Updating files: 97% (3511/3619) Updating files: 98% (3547/3619) Updating files: 99% (3583/3619) Updating files: 100% (3619/3619) Updating files: 100% (3619/3619), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:b04c81bb61f199e1c61b3fdfba3dace3ec16df3ef11d647f4d7e4fa3fd316f57 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:b04c81bb61f199e1c61b3fdfba3dace3ec16df3ef11d647f4d7e4fa3fd316f57 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:9dffcf6b4f5a14360ebd0f43f213ea18d56a17901f1c024e8af96b4bf39a29e1 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:9dffcf6b4f5a14360ebd0f43f213ea18d56a17901f1c024e8af96b4bf39a29e1 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:77aba231a4f935ae9f49d1c4a2cf0249996866aef1c76b78ee65f8f8b2831c77 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:77aba231a4f935ae9f49d1c4a2cf0249996866aef1c76b78ee65f8f8b2831c77 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:fef7b07c289cb917d31f1e5e8a17e262702d11fd6d58050c48ea516cf899120d [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:fef7b07c289cb917d31f1e5e8a17e262702d11fd6d58050c48ea516cf899120d [e2e-llm-inference-service] + make e2e-llmisvc-ocp RUNNING_LOCAL=false [e2e-llm-inference-service] /bin/sh: line 1: go: command not found [e2e-llm-inference-service] make run-e2e-ocp E2E_MARKER="llmisvc_core and cluster_cpu and not pvc_storage" E2E_PARALLELISM=2 E2E_PROFILE=llm-d SETUP_E2E=true [e2e-llm-inference-service] make[1]: Entering directory '/workspace/source' [e2e-llm-inference-service] /bin/sh: line 1: go: command not found [e2e-llm-inference-service] SETUP_E2E="true" \ [e2e-llm-inference-service] OPERATOR_TYPE="" \ [e2e-llm-inference-service] KSERVE_NAMESPACE="kserve" \ [e2e-llm-inference-service] RUNNING_LOCAL="false" \ [e2e-llm-inference-service] PYTEST_ARGS="" \ [e2e-llm-inference-service] ./test/scripts/openshift-ci/run-e2e-tests.sh "llmisvc_core and cluster_cpu and not pvc_storage" "2" "llm-d" [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.34) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1969 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] OPT_125M_OCI_MODEL_URI=oci://quay.io/opendatahub/opt-125m-modelcar:latest [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1969 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1969 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make[2]: Entering directory '/workspace/source' [e2e-llm-inference-service] /bin/sh: line 1: go: command not found [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make[2]: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 301 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading setuptools (1.0MiB) [e2e-llm-inference-service] Downloading pyarrow (47.7MiB) [e2e-llm-inference-service] Downloading kubernetes (4.4MiB) [e2e-llm-inference-service] Downloading grpcio (6.9MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading numpy (16.1MiB) [e2e-llm-inference-service] Downloading pandas (12.2MiB) [e2e-llm-inference-service] Downloading pillow (6.6MiB) [e2e-llm-inference-service] Downloading pygments (1.2MiB) [e2e-llm-inference-service] Downloading uvloop (3.6MiB) [e2e-llm-inference-service] Downloading botocore (15.1MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading portforward (4.0MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.6MiB) [e2e-llm-inference-service] Downloading black (1.8MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading pygments [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading pillow [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 116 packages in 2.11s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 116 packages in 296ms [e2e-llm-inference-service] + aiohappyeyeballs==2.7.1 [e2e-llm-inference-service] + aiohttp==3.14.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.5 [e2e-llm-inference-service] + annotated-types==0.8.0 [e2e-llm-inference-service] + anyio==4.15.1 [e2e-llm-inference-service] + asgiref==3.12.1 [e2e-llm-inference-service] + attrs==26.1.0 [e2e-llm-inference-service] + avro==1.12.2 [e2e-llm-inference-service] + black==26.5.1 [e2e-llm-inference-service] + boto3==1.43.93 [e2e-llm-inference-service] + botocore==1.43.93 [e2e-llm-inference-service] + certifi==2026.7.22 [e2e-llm-inference-service] + cffi==2.1.1 [e2e-llm-inference-service] + charset-normalizer==3.5.1 [e2e-llm-inference-service] + click==8.5.0 [e2e-llm-inference-service] + cloudevents==1.13.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.12.0 [e2e-llm-inference-service] + coverage==7.16.1 [e2e-llm-inference-service] + cryptography==50.0.1 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.11 [e2e-llm-inference-service] + execnet==2.1.2 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.8.0 [e2e-llm-inference-service] + googleapis-common-protos==1.75.3 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.84.0 [e2e-llm-inference-service] + grpcio-testing==1.81.1 [e2e-llm-inference-service] + grpcio-tools==1.81.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.8.0 [e2e-llm-inference-service] + httpx==0.28.1 [e2e-llm-inference-service] + httpx-retries==0.6.0 [e2e-llm-inference-service] + idna==3.19 [e2e-llm-inference-service] + iniconfig==2.3.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.1.0 [e2e-llm-inference-service] + kserve==0.21.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==36.0.3 [e2e-llm-inference-service] + markupsafe==3.0.3 [e2e-llm-inference-service] + multidict==6.8.0 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.1.0 [e2e-llm-inference-service] + numpy==2.3.5 [e2e-llm-inference-service] + oauthlib==3.3.1 [e2e-llm-inference-service] + opentelemetry-api==1.44.0 [e2e-llm-inference-service] + opentelemetry-exporter-otlp==1.44.0 [e2e-llm-inference-service] + opentelemetry-exporter-otlp-proto-common==1.44.0 [e2e-llm-inference-service] + opentelemetry-exporter-otlp-proto-grpc==1.44.0 [e2e-llm-inference-service] + opentelemetry-exporter-otlp-proto-http==1.44.0 [e2e-llm-inference-service] + opentelemetry-instrumentation==0.65b0 [e2e-llm-inference-service] + opentelemetry-instrumentation-asgi==0.65b0 [e2e-llm-inference-service] + opentelemetry-instrumentation-fastapi==0.65b0 [e2e-llm-inference-service] + opentelemetry-instrumentation-grpc==0.65b0 [e2e-llm-inference-service] + opentelemetry-instrumentation-logging==0.65b0 [e2e-llm-inference-service] + opentelemetry-proto==1.44.0 [e2e-llm-inference-service] + opentelemetry-sdk==1.44.0 [e2e-llm-inference-service] + opentelemetry-semantic-conventions==0.65b0 [e2e-llm-inference-service] + opentelemetry-util-http==0.65b0 [e2e-llm-inference-service] + orjson==3.12.0 [e2e-llm-inference-service] + packaging==26.3 [e2e-llm-inference-service] + pandas==2.3.3 [e2e-llm-inference-service] + pathspec==1.1.1 [e2e-llm-inference-service] + pillow==12.3.0 [e2e-llm-inference-service] + platformdirs==4.11.8 [e2e-llm-inference-service] + pluggy==1.6.0 [e2e-llm-inference-service] + portforward==0.7.6 [e2e-llm-inference-service] + prometheus-client==0.26.0 [e2e-llm-inference-service] + propcache==0.5.2 [e2e-llm-inference-service] + protobuf==6.33.6 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==25.0.1 [e2e-llm-inference-service] + pyasn1==0.6.4 [e2e-llm-inference-service] + pycparser==3.0 [e2e-llm-inference-service] + pydantic==2.13.5 [e2e-llm-inference-service] + pydantic-core==2.46.5 [e2e-llm-inference-service] + pygments==2.21.0 [e2e-llm-inference-service] + pyjwt==2.14.0 [e2e-llm-inference-service] + pytest==9.1.1 [e2e-llm-inference-service] + pytest-asyncio==1.4.0 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.36.2 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.8.0 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.2.3 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytokens==0.4.1 [e2e-llm-inference-service] + pytz==2026.3.post1 [e2e-llm-inference-service] + pyyaml==6.0.3 [e2e-llm-inference-service] + requests==2.34.2 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + s3transfer==0.19.2 [e2e-llm-inference-service] + setuptools==80.10.2 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + starlette==1.6.0 [e2e-llm-inference-service] + tabulate==0.10.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.2 [e2e-llm-inference-service] + tomlkit==0.15.1 [e2e-llm-inference-service] + typing-extensions==4.16.0 [e2e-llm-inference-service] + typing-inspection==0.4.4 [e2e-llm-inference-service] + tzdata==2026.4 [e2e-llm-inference-service] + urllib3==2.7.0 [e2e-llm-inference-service] + uvicorn==0.53.0 [e2e-llm-inference-service] + uvloop==0.22.1 [e2e-llm-inference-service] + watchfiles==1.2.0 [e2e-llm-inference-service] + websocket-client==1.9.2 [e2e-llm-inference-service] + websockets==17.1 [e2e-llm-inference-service] + wrapt==2.4.1 [e2e-llm-inference-service] + yarl==1.24.5 [e2e-llm-inference-service] Audited 1 package in 49ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make[2]: Entering directory '/workspace/source' [e2e-llm-inference-service] /bin/sh: line 1: go: command not found [e2e-llm-inference-service] make[2]: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-3 found, but not yet Succeeded (Phase: Pending). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-3 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-3 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-3 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-3 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-3 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-579c8958f7-kl4cx 1/1 Running 0 40s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-579c8958f7-kl4cx condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-77857b7b98-66mqh 1/1 Running 0 44s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-77857b7b98-66mqh condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-6c4d864b86-2p69k 1/1 Running 0 49s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-6c4d864b86-2p69k condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.34) [e2e-llm-inference-service] 🎯 Server version (4.21.34) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-hrsdw 1/1 Running 0 13s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-hrsdw condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-7b7995d5cd-9jj29 0/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-7b7995d5cd-9jj29 condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-8699b59bbc-s8kvl 0/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.3 found, but not yet Succeeded (Phase: Pending). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.3 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.3 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.3 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.3 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.3 found, but not yet Succeeded (Phase: Installing). Waiting... (50/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.3 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6fc8f864f4-4srxn 1/1 Running 0 84s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6fc8f864f4-4srxn condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6fc8f864f4-4srxn 1/1 Running 0 93s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6fc8f864f4-4srxn condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Now using project "kserve" on server "https://abc95519c67f9460189748464d737b7a-a75d8cad7bdd6449.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make[2]: Entering directory '/workspace/source' [e2e-llm-inference-service] /bin/sh: line 1: go: command not found [e2e-llm-inference-service] make[2]: Leaving directory '/workspace/source' [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:9dffcf6b4f5a14360ebd0f43f213ea18d56a17901f1c024e8af96b4bf39a29e1 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:fef7b07c289cb917d31f1e5e8a17e262702d11fd6d58050c48ea516cf899120d [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:b04c81bb61f199e1c61b3fdfba3dace3ec16df3ef11d647f4d7e4fa3fd316f57 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:77aba231a4f935ae9f49d1c4a2cf0249996866aef1c76b78ee65f8f8b2831c77 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:9dffcf6b4f5a14360ebd0f43f213ea18d56a17901f1c024e8af96b4bf39a29e1 [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:fef7b07c289cb917d31f1e5e8a17e262702d11fd6d58050c48ea516cf899120d [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:b04c81bb61f199e1c61b3fdfba3dace3ec16df3ef11d647f4d7e4fa3fd316f57 [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:77aba231a4f935ae9f49d1c4a2cf0249996866aef1c76b78ee65f8f8b2831c77 [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] kserve-llm-d-tokenizer=registry.redhat.io/rhaii/vllm-cpu-rhel9@sha256:0ec0f5ff9a5ff263f58db0cfb19e3d7eef24b94c2d8629fe5145f5a8747e28c9 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-latency-predictor-prediction=quay.io/opendatahub/odh-latency-predictor-prediction:odh-stable [e2e-llm-inference-service] kserve-llm-d-latency-predictor-training=quay.io/opendatahub/odh-latency-predictor-training:odh-stable [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-master [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-master [e2e-llm-inference-service] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-eppconfig-default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-eppconfig-default-pd serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/clusterservingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-7774f8b66-9w5pn 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-7774f8b66-9w5pn condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterservingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-eppconfig-default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-eppconfig-default-pd serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/clusterservingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-575575dfd7-mvh9l" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-575575dfd7-nrvzj condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-openshift-distro-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-openshift-distro-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] networkpolicy.networking.k8s.io/odh-model-controller-webhook-allow-ingress created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-mlserver created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-mlserver-cuda created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-ovms created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu-x86 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu-x86-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cpu-x86-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cuda created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cuda-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-cuda-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-gaudi created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-gaudi-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-gaudi-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-rocm created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-rocm-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-rocm-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-ppc64le created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-ppc64le-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-ppc64le-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-s390x created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-s390x-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-s390x-fast-2 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-x86 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-x86-fast-1 created [e2e-llm-inference-service] clusterservingruntime.serving.kserve.io/csr-kserve-vllm-spyre-x86-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-omni-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-omni-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-omni-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-llm-sglang created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.34 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.34 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.34 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.3 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.2 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.2 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.3 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.3 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-3 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.2 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.2 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.3 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.3 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.2 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.2 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.3 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.3 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.2 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.2 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.3 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.3 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.2 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.2 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.3 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.3 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.2 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.2 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.3 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:9dffcf6b4f5a14360ebd0f43f213ea18d56a17901f1c024e8af96b4bf39a29e1 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:5611bba976b64913230435e8cf2d4ba90dd00e3de8dcf0e7c25ee7460796a5e3 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:7eee98850d889006e68133cff0ac0fed10b51eb61b8dffcbecf16c8e3f2f0d37 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:fef7b07c289cb917d31f1e5e8a17e262702d11fd6d58050c48ea516cf899120d [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:43123f2221c4d9efd86e4ece9a85bb1ee46b8b7f552f53b0061a6337e877ef50 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llmisvc_core and cluster_cpu and not pvc_storage [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-9.1.1, pluggy-1.6.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.134.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '9.1.1', 'pluggy': '1.6.0'}, 'Plugins': {'asyncio': '1.4.0', 'xdist': '3.8.0', 'anyio': '4.15.1', 'cov': '5.0.0', 'httpx': '0.36.2', 'json-report': '1.5.0', 'metadata': '3.1.1'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: asyncio-1.4.0, xdist-3.8.0, anyio-4.15.1, cov-5.0.0, httpx-0.36.2, json-report-1.5.0, metadata-3.1.1 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [40 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] 2026-09-18 22:04:55.012 6511 kserve INFO [conftest.py:configure_logger():40] trace_id=none Logger configured [e2e-llm-inference-service] 2026-09-18 22:04:55.012 6508 kserve INFO [conftest.py:configure_logger():40] trace_id=none Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-09-18 22:21:53.283 6508 kserve.trace trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-09-18 22:21:53.283 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-09-18 22:21:53.312 6508 kserve.trace trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:21:53.312 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:21:53.321 6508 kserve.trace trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:21:53.321 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-09-18 22:22:21.156 6508 kserve.trace trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-09-18 22:22:21.156 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-09-18 22:22:21.186 6508 kserve.trace trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:22:21.186 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:22:21.192 6508 kserve.trace trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:22:21.192 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-09-18 22:32:43.062 6508 kserve.trace trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-6f0aa991 [e2e-llm-inference-service] 2026-09-18 22:32:43.062 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-6f0aa991 [e2e-llm-inference-service] 2026-09-18 22:32:43.094 6508 kserve.trace trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:32:43.094 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:32:43.103 6508 kserve.trace trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:32:43.103 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-oci] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-oci] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-template-args-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-template-args-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-09-18 22:45:13.415 6508 kserve.trace trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-09-18 22:45:13.415 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway router-gateway-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-09-18 22:45:13.439 6508 kserve.trace trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:45:13.439 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:45:13.447 6508 kserve.trace trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:45:13.447 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-09-18 22:45:13.448 6508 kserve.trace trace_id=none Checking HttpRoute router-route-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-09-18 22:45:13.448 6508 kserve.trace INFO [gw_api.py:create_or_update_route():121] trace_id=none Checking HttpRoute router-route-1 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-09-18 22:45:13.451 6508 kserve.trace trace_id=none Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-09-18 22:45:13.451 6508 kserve.trace INFO [gw_api.py:create_or_update_route():149] trace_id=none Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-09-18 22:45:13.463 6508 kserve.trace trace_id=none ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-09-18 22:45:13.463 6508 kserve.trace INFO [gw_api.py:create_or_update_route():157] trace_id=none ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-09-18 22:45:13.463 6508 kserve.trace trace_id=none Checking HttpRoute router-route-2 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-09-18 22:45:13.463 6508 kserve.trace INFO [gw_api.py:create_or_update_route():121] trace_id=none Checking HttpRoute router-route-2 in namespace e2e-test-llm-inference-service-028f7809 [e2e-llm-inference-service] 2026-09-18 22:45:13.467 6508 kserve.trace trace_id=none Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-09-18 22:45:13.467 6508 kserve.trace INFO [gw_api.py:create_or_update_route():149] trace_id=none Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-09-18 22:45:13.476 6508 kserve.trace trace_id=none ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-09-18 22:45:13.476 6508 kserve.trace INFO [gw_api.py:create_or_update_route():157] trace_id=none ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-09-18 22:55:14.720 6508 kserve.trace trace_id=none Checking Gateway router-gateway-2 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-09-18 22:55:14.720 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] trace_id=none Checking Gateway router-gateway-2 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-09-18 22:55:14.759 6508 kserve.trace trace_id=none Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-09-18 22:55:14.759 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] trace_id=none Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-09-18 22:55:14.767 6508 kserve.trace trace_id=none ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-09-18 22:55:14.767 6508 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] trace_id=none ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-09-18 22:55:14.768 6508 kserve.trace trace_id=none Checking HttpRoute router-route-3 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-09-18 22:55:14.768 6508 kserve.trace INFO [gw_api.py:create_or_update_route():121] trace_id=none Checking HttpRoute router-route-3 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-09-18 22:55:14.773 6508 kserve.trace trace_id=none Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-09-18 22:55:14.773 6508 kserve.trace INFO [gw_api.py:create_or_update_route():149] trace_id=none Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-09-18 22:55:14.792 6508 kserve.trace trace_id=none ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-09-18 22:55:14.792 6508 kserve.trace INFO [gw_api.py:create_or_update_route():157] trace_id=none ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-09-18 22:55:14.792 6508 kserve.trace trace_id=none Checking HttpRoute router-route-4 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-09-18 22:55:14.792 6508 kserve.trace INFO [gw_api.py:create_or_update_route():121] trace_id=none Checking HttpRoute router-route-4 in namespace e2e-test-llm-inference-service-b5dd93e6 [e2e-llm-inference-service] 2026-09-18 22:55:14.796 6508 kserve.trace trace_id=none Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-09-18 22:55:14.796 6508 kserve.trace INFO [gw_api.py:create_or_update_route():149] trace_id=none Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-09-18 22:55:14.813 6508 kserve.trace trace_id=none ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-09-18 22:55:14.813 6508 kserve.trace INFO [gw_api.py:create_or_update_route():157] trace_id=none ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-pvc] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters-one-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters-one-pvc] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] __________ test_flow_control_smoke[flow-control-concurrency-detector] __________ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-flow-control-concurrency-detector', 'workload-llmd-simulator'], promp... {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] flow_control_auth = None [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-flow-control-round-robin", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="fc-smoke-test", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.flow_control, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="flow-control-utilization-detector", [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-flow-control-concurrency-detector", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="fc-concurrency-test", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.flow_control, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="flow-control-concurrency-detector", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=True, [e2e-llm-inference-service] ) [e2e-llm-inference-service] def test_flow_control_smoke(test_case: TestCase, flow_control_auth): [e2e-llm-inference-service] """Verify that the EPP boots and serves traffic with flow control enabled. [e2e-llm-inference-service] [e2e-llm-inference-service] Sends requests with different fairness IDs, InferenceObjective headers, and [e2e-llm-inference-service] default headers. When a downstream auth provider is available (via the [e2e-llm-inference-service] flow_control_auth fixture), also verifies the auth -> flow control pipeline. [e2e-llm-inference-service] """ [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] [e2e-llm-inference-service] if flow_control_auth: [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] test_case.llm_service.metadata.annotations.update( [e2e-llm-inference-service] flow_control_auth.get("annotations", {}) [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py:122: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...curre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {} [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....ol-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1432: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f4c962dd800> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] ^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1443: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T22:07:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1427: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-flow-control-smoke-1e98f46d with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig router-managed-fc-concurrency-t-25c29f3a in namespace e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig router-managed-fc-concurrency-t-25c29f3a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig router-managed-fc-concurrency-t-25c29f3a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig scheduler-flow-control-concurre-34e50aeb in namespace e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig scheduler-flow-control-concurre-34e50aeb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig scheduler-flow-control-concurre-34e50aeb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig workload-llmd-simulator-fc-conc-1c5a0c75 in namespace e2e-test-flow-control-smoke-1e98f46d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-fc-conc-1c5a0c75 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-fc-conc-1c5a0c75 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-09-18T22:06:51.905329] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-flow-control-smoke-1e98f46d', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-09-18T22:06:52.153361] end - ✅ in 0.248s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-09-18T22:06:52.153508] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'fc-concurrency-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-flow-control-smoke-1e98f46d', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-fc-concurrency-t-25c29f3a'}, [e2e-llm-inference-service] {'name': 'scheduler-flow-control-concurre-34e50aeb'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-fc-conc-1c5a0c75'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T22:07:25Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-route]', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'message': 'Inference Pool e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-route]', 'reason': 'WaitingForGateway', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-route]', 'reason': 'WaitingForGateway', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T22:07:39Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T22:07:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:07:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:07:42Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 1: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-09-18T22:05:20Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 2: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-09-18T22:05:20Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T22:07:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 3: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-09-18T22:05:20Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1447 Timed out waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T22:07:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-09-18T22:21:52.853156] end - ❌ 900.699s: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T22:07:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_flow_control.py:171 [router-managed-scheduler-flow-control-concurrency-detector-workload-llmd-simulator] Failed: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T22:07:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:190 Skipping deletion of namespace e2e-test-flow-control-smoke-1e98f46d (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KSer... {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-oci", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Same migration path, but with the legacy plugin config supplied via [e2e-llm-inference-service] # spec.router.scheduler.template.containers[].args instead of [e2e-llm-inference-service] # spec.router.scheduler.config.inline. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-template-args-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-template-args-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:922: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...grati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {} [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....fig-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1432: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f9a5eeda980> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] ^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1443: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:08:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1427: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-9c386aa9 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 in namespace e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 in namespace e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b in namespace e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-09-18T22:08:27.926966] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KServe is a', service_name='scheduler-v06-pd-migration-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-9c386aa9', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-9c386aa9', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-09-18T22:08:27.939904] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-9c386aa9', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-09-18T22:08:28.058305] end - ✅ in 0.118s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-09-18T22:08:28.058400] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-9c386aa9', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:08:36Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-route]', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:08:36Z', 'message': 'Inference Pool e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:08:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:36Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:36Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:08:36Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-route]', 'reason': 'WaitingForGateway', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:08:36Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-route]', 'reason': 'WaitingForGateway', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:08:36Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:08:46Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:08:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:08:48Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:08:48Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:08:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1447 Timed out waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:08:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-09-18T22:23:28.868986] end - ❌ 900.810s: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:08:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:959 [router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] ❌ ERROR: Failed to call llm inference service scheduler-v06-pd-migration-test: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:08:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:274 # Diagnostics for 'scheduler-v06-pd-migration-test' in 'e2e-test-llm-inference-service-9c386aa9' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:275 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:278 # LLMInferenceService scheduler-v06-pd-migration-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:287 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:27Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:27Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:28Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-eppconfig-default: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-eppconfig-default-pd: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-sglang-template: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:18:34Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] resourceVersion: '31687' [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] - name: scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] - name: workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-eppconfig-default: kserve-config-llm-scheduler-eppconfig-default [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-eppconfig-default-pd: kserve-config-llm-scheduler-eppconfig-default-pd [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-sglang-template: kserve-config-sglang-template [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:48Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:18:34Z' [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:18:34Z' [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:18:34Z' [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-9c386aa9 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9 to ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:34 e2e-test-llm-inference-service-9c386aa9 multus Normal AddedInterface Add eth0 [10.133.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:34 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-142-209.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:35 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-142-209.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0" in 1.547s (1.547s including waiting). Image size: 170636348 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:35 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-142-209.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:35 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-142-209.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:36 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-142-209.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:36 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-142-209.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:36 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-142-209.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-9c386aa9 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk to ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:34 e2e-test-llm-inference-service-9c386aa9 multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:34 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-141-13.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:35 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-141-13.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" in 1.596s (1.596s including waiting). Image size: 79178094 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:36 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-141-13.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:36 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-141-13.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-prefill-8665687897 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-9c386aa9 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp to ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:34 e2e-test-llm-inference-service-9c386aa9 multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:19:36 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-141-13.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:19:36 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-141-13.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:19:36 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-141-13.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:18:44 e2e-test-llm-inference-service-9c386aa9 kubelet/ip-10-0-141-13.ec2.internal Warning BackOff Back-off restarting failed container main in pod scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-5c479b44c4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:32 e2e-test-llm-inference-service-9c386aa9 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-pd-migration-test-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-pd-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:33 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-d3853b0b1e7c3750d40ebaedb563580b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:46 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:45 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:08:45 e2e-test-llm-inference-service-9c386aa9 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:252 ### EndpointSlice scheduler-v06-pd-migration-test-epp-service-q4hgh (service=scheduler-v06-pd-migration-test-epp-service): [{'addresses': ['10.134.0.34'], 'ready': False, 'targetRef': 'scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:252 ### EndpointSlice scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf5k5tz (service=scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf): [{'addresses': ['10.133.0.44'], 'ready': True, 'targetRef': 'scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9'}, {'addresses': ['10.134.0.33'], 'ready': True, 'targetRef': 'scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:252 ### EndpointSlice scheduler-v06-pd-migration-test-kserve-workload-svc-lqptq (service=scheduler-v06-pd-migration-test-kserve-workload-svc): [{'addresses': ['10.134.0.33'], 'ready': True, 'targetRef': 'scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk'}, {'addresses': ['10.133.0.44'], 'ready': True, 'targetRef': 'scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 b'{"severity_text":"INFO","timestamp":"2026-09-18T22:08:35.963631812Z","body":"Proxy starting","service.name":"llm-d-router-disagg-sidecar","Built on":"","From Git SHA":"unknown","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:35.963660153Z","body":"Proxy configuration","service.name":"llm-d-router-disagg-sidecar","config":"{\\"Port\\":\\"8000\\",\\"KVConnector\\":\\"nixlv2\\",\\"ECConnector\\":\\"\\",\\"DataParallelSize\\":1,\\"MaxIdleConnsPerHost\\":1024,\\"EnablePrefillerSampling\\":false,\\"PrefillMaxRetries\\":0,\\"PrefillRetryBackoff\\":200000000,\\"UseTLSForPrefiller\\":true,\\"UseTLSForDecoder\\":true,\\"UseTLSForEncoder\\":false,\\"InsecureSkipVerifyForPrefiller\\":false,\\"InsecureSkipVerifyForEncoder\\":false,\\"InsecureSkipVerifyForDecoder\\":false,\\"SecureServing\\":true,\\"CertPath\\":\\"/var/run/kserve/tls\\",\\"TLSMinVersion\\":0,\\"TLSCipherSuites\\":null,\\"MetricsPort\\":0,\\"MetricsCertDir\\":\\"\\",\\"MooncakeBootstrapPort\\":8998,\\"P2PConnectorPort\\":7777,\\"EnableP2PPull\\":false,\\"EnableSSRFProtection\\":true,\\"InferencePoolNamespace\\":\\"e2e-test-llm-inference-service-9c386aa9\\",\\"InferencePoolName\\":\\"scheduler-v06-pd-migration-test-inference-pool\\",\\"PoolGroup\\":\\"inference.networking.x-k8s.io\\",\\"DecodeChunkSize\\":0,\\"Tracing\\":false,\\"MoRIIOWriteMode\\":false,\\"MoRIIODecodeNotifyPort\\":61005,\\"MoRIIODecodeHandshakePort\\":6301,\\"MoRIIODecodePodIP\\":\\"\\",\\"MoRIIOParallelDispatch\\":false,\\"MoRIIOParallelDecodeWaitTimeout\\":30000000000,\\"MoRIIOPrefillHandshakePort\\":6301,\\"MoRIIOPrefillNotifyPort\\":61005,\\"MoRIIOTPSize\\":1,\\"MoRIIODPSize\\":1,\\"MoRIIORemoteHosts\\":null,\\"MoRIIODPSizeLocal\\":0,\\"MoRIIODecodeHosts\\":null,\\"MoRIIORemoteHostSpecs\\":null,\\"MoRIIODecodeHostSpecs\\":null,\\"MoRIIODecodePodIPSpec\\":\\"\\",\\"DecoderURL\\":\\"https://localhost:8001\\"}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:35.964913865Z","logger":"allowlist-validator","body":"starting SSRF protection allowlist validator","service.name":"llm-d-router-disagg-sidecar","namespace":"e2e-test-llm-inference-service-9c386aa9","poolName":"scheduler-v06-pd-migration-test-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:35.976212721Z","logger":"allowlist-validator","body":"InferencePool added","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:35.981827861Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":0,"targets":{},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:35.981898819Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":0,"targets":{},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:36.065938602Z","logger":"allowlist-validator","body":"allowlist validator started successfully","service.name":"llm-d-router-disagg-sidecar","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:36.067158785Z","logger":"proxy server on port 8000","body":"server TLS configured","service.name":"llm-d-router-disagg-sidecar","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:36.067171702Z","logger":"proxy server on port 8000","body":"starting","service.name":"llm-d-router-disagg-sidecar","addr":"[::]:8000","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:36.134270201Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":2,"targets":{"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:36.145213707Z","logger":"allowlist- [e2e-llm-inference-service] validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:37.156185107Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:44.07884151Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:44.139762291Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:44.177911594Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:44.201045663Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:08:45.209289916Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:09:05.976927408Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:09:05.985427865Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:09:05.985489657Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:09:35.978003442Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:09:35.985150911Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.1 [e2e-llm-inference-service] 33.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:09:35.985202035Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:10:05.978104388Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:10:05.985925189Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:10:05.985992239Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:10:35.978978072Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:10:35.986470052Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:10:35.986532451Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:11:05.979025357Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:11:05.986715204Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:11:05.98676327Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:11:35.979263499Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:11:35.987360588Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets" [e2e-llm-inference-service] :{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:11:35.987407081Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:12:05.979710734Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:12:05.987171009Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:12:05.987220291Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:12:35.980476749Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:12:35.987333121Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:12:35.987385991Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:13:05.981254297Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:13:05.989152999Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:13:05.989206412Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:13:35.981428135Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:13:35.990567741Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4," [e2e-llm-inference-service] targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:13:35.990616149Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:14:05.981547642Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:14:05.989697072Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:14:05.989752619Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:14:35.982585421Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:14:35.989597907Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:14:35.98964761Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:15:05.982619192Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:15:05.991680702Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:15:05.991744009Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:15:35.9828966Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:15:35.991982939Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount [e2e-llm-inference-service] ":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:15:35.992053108Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:16:05.983092315Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:16:05.991332671Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:16:05.991383297Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:16:35.98358553Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:16:35.990826491Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:16:35.990876868Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:17:05.984150394Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:17:05.992794767Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:17:05.992855628Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:17:35.984970194Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:17:35.992803531Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targ [e2e-llm-inference-service] etCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:17:35.992864538Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:18:05.985781441Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:18:05.994393888Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:18:05.994449311Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:18:35.986511341Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:18:35.99395865Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:18:35.994005677Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:05.987225642Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:05.994813548Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:05.994873365Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:35.987417316Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:35.994884128Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar [e2e-llm-inference-service] ","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:35.994937338Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:20:05.987858343Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:20:05.99522055Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:20:05.995269602Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:20:35.988791038Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:20:35.997770699Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:20:35.997828976Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:21:05.989265827Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:21:05.996983406Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:21:05.997052579Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:21:35.989343379Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:21:35.997403702Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg- [e2e-llm-inference-service] sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:21:35.997473225Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:22:05.990287121Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:22:05.997805632Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:22:05.997850932Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:22:35.990670893Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:22:35.998626681Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:22:35.998685637Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:05.991608981Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-pd-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:05.999447278Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:05.999496961Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.133.0.44":{},"10.134.0.33":{},"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9":{},"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk":{}},"severity_number":9}\n' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 b'I0918 22:08:36.210218 1 config.go:808] "Configuration:" =<\n\t{\n\t "DisplayModelName": "facebook/opt-125m",\n\t "IP": "",\n\t "PodName": "",\n\t "PodNameSpace": "",\n\t "VllmDevMode": false,\n\t "block-size": 16,\n\t "data-parallel-rank": -1,\n\t "data-parallel-size": 1,\n\t "dataset-in-memory": false,\n\t "dataset-path": "",\n\t "dataset-table-name": "llmd",\n\t "dataset-url": "",\n\t "default-embedding-dimensions": 384,\n\t "ec-transfer-config": "",\n\t "enable-kvcache": false,\n\t "enable-prefix-caching": false,\n\t "enable-request-id-headers": false,\n\t "enable-sleep-mode": false,\n\t "enforce-eager": false,\n\t "event-batch-size": 16,\n\t "failure-injection-rate": 0,\n\t "failure-types": null,\n\t "fake-metrics": null,\n\t "fake-metrics-refresh-interval": "100ms",\n\t "force-dummy-tokenizer": true,\n\t "global-cache-hit-threshold": 0,\n\t "hash-seed": "",\n\t "image-emission-rate": 0,\n\t "inter-token-latency": "100ms",\n\t "inter-token-latency-std-dev": "0s",\n\t "kv-cache-size": 1024,\n\t "kv-cache-transfer-latency": "0s",\n\t "kv-cache-transfer-latency-std-dev": "0s",\n\t "kv-cache-transfer-time-per-token": "0s",\n\t "kv-cache-transfer-time-std-dev": "0s",\n\t "latency-calculator": "",\n\t "log-http": false,\n\t "lora-modules": null,\n\t "max-cpu-loras": 1,\n\t "max-loras": 1,\n\t "max-model-len": 1024,\n\t "max-num-seqs": 5,\n\t "max-request-body-size-mb": 4,\n\t "max-tool-call-array-param-length": 5,\n\t "max-tool-call-integer-param": 100,\n\t "max-tool-call-number-param": 100,\n\t "max-waiting-queue-length": 1000,\n\t "min-tool-call-array-param-length": 1,\n\t "min-tool-call-integer-param": 0,\n\t "min-tool-call-number-param": 0,\n\t "mm-encoder-only": false,\n\t "mm-processor-kwargs": "",\n\t "mm-render-timeout": "1m0s",\n\t "mode": "random",\n\t "model": "facebook/opt-125m",\n\t "object-tool-call-not-required-field-probability": 50,\n\t "omni": false,\n\t "port": 8001,\n\t "prefill-overhead": "0s",\n\t "prefill-time-per-token": "0s",\n\t "prefill-time-std-dev": "0s",\n\t "render-timeout": "30s",\n\t "render-url": "http://localhost:8082",\n\t "seed": 1789769316209721600,\n\t "self-signed-certs": false,\n\t "served-model-name": [\n\t "facebook/opt-125m"\n\t ],\n\t "ssl-certfile": "/var/run/kserve/tls/tls.crt",\n\t "ssl-keyfile": "/var/run/kserve/tls/tls.key",\n\t "startup-duration": "0s",\n\t "time-factor-under-load": 1,\n\t "time-to-first-token": "200ms",\n\t "time-to-first-token-std-dev": "0s",\n\t "time-to-generate-image": "0s",\n\t "time-to-generate-image-std-dev": "0s",\n\t "tool-call-not-required-param-probability": 50,\n\t "use-vllm-map-event-format": false,\n\t "zmq-endpoint": "tcp://127.0.0.1:5557"\n\t}\n >\nI0918 22:08:36.210282 1 tokenizer.go:60] "Force dummy tokenizer flag is set, using simulated tokenizer" model="facebook/opt-125m"\nI0918 22:08:36.214268 1 context.go:216] "No dataset path or URL provided, using random text for responses"\nI0918 22:08:36.214348 1 communication.go:66] "Starting communication layer"\nI0918 22:08:36.214381 1 simulator.go:210] "Start processing routine"\nI0918 22:08:36.214685 1 http_server_tls.go:59] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key"\nI0918 22:08:36.214757 1 grpc.go:129] "Server starting" protocol="gRPC" port=8001\nI0918 22:08:36.215373 1 http.go:117] "Server starting" protocol="HTTPS" port=8001\n' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 b'I0918 22:08:36.074528 1 config.go:808] "Configuration:" =<\n\t{\n\t "DisplayModelName": "facebook/opt-125m",\n\t "IP": "",\n\t "PodName": "",\n\t "PodNameSpace": "",\n\t "VllmDevMode": false,\n\t "block-size": 16,\n\t "data-parallel-rank": -1,\n\t "data-parallel-size": 1,\n\t "dataset-in-memory": false,\n\t "dataset-path": "",\n\t "dataset-table-name": "llmd",\n\t "dataset-url": "",\n\t "default-embedding-dimensions": 384,\n\t "ec-transfer-config": "",\n\t "enable-kvcache": false,\n\t "enable-prefix-caching": false,\n\t "enable-request-id-headers": false,\n\t "enable-sleep-mode": false,\n\t "enforce-eager": false,\n\t "event-batch-size": 16,\n\t "failure-injection-rate": 0,\n\t "failure-types": null,\n\t "fake-metrics": null,\n\t "fake-metrics-refresh-interval": "100ms",\n\t "force-dummy-tokenizer": true,\n\t "global-cache-hit-threshold": 0,\n\t "hash-seed": "",\n\t "image-emission-rate": 0,\n\t "inter-token-latency": "0s",\n\t "inter-token-latency-std-dev": "0s",\n\t "kv-cache-size": 1024,\n\t "kv-cache-transfer-latency": "0s",\n\t "kv-cache-transfer-latency-std-dev": "0s",\n\t "kv-cache-transfer-time-per-token": "0s",\n\t "kv-cache-transfer-time-std-dev": "0s",\n\t "latency-calculator": "",\n\t "log-http": false,\n\t "lora-modules": null,\n\t "max-cpu-loras": 1,\n\t "max-loras": 1,\n\t "max-model-len": 1024,\n\t "max-num-seqs": 5,\n\t "max-request-body-size-mb": 4,\n\t "max-tool-call-array-param-length": 5,\n\t "max-tool-call-integer-param": 100,\n\t "max-tool-call-number-param": 100,\n\t "max-waiting-queue-length": 1000,\n\t "min-tool-call-array-param-length": 1,\n\t "min-tool-call-integer-param": 0,\n\t "min-tool-call-number-param": 0,\n\t "mm-encoder-only": false,\n\t "mm-processor-kwargs": "",\n\t "mm-render-timeout": "1m0s",\n\t "mode": "random",\n\t "model": "facebook/opt-125m",\n\t "object-tool-call-not-required-field-probability": 50,\n\t "omni": false,\n\t "port": 8000,\n\t "prefill-overhead": "0s",\n\t "prefill-time-per-token": "0s",\n\t "prefill-time-std-dev": "0s",\n\t "render-timeout": "30s",\n\t "render-url": "http://localhost:8082",\n\t "seed": 1789769316074012700,\n\t "self-signed-certs": false,\n\t "served-model-name": [\n\t "facebook/opt-125m"\n\t ],\n\t "ssl-certfile": "/var/run/kserve/tls/tls.crt",\n\t "ssl-keyfile": "/var/run/kserve/tls/tls.key",\n\t "startup-duration": "0s",\n\t "time-factor-under-load": 1,\n\t "time-to-first-token": "5s",\n\t "time-to-first-token-std-dev": "0s",\n\t "time-to-generate-image": "0s",\n\t "time-to-generate-image-std-dev": "0s",\n\t "tool-call-not-required-param-probability": 50,\n\t "use-vllm-map-event-format": false,\n\t "zmq-endpoint": "tcp://127.0.0.1:5557"\n\t}\n >\nI0918 22:08:36.074614 1 tokenizer.go:60] "Force dummy tokenizer flag is set, using simulated tokenizer" model="facebook/opt-125m"\nI0918 22:08:36.080745 1 context.go:216] "No dataset path or URL provided, using random text for responses"\nI0918 22:08:36.080804 1 communication.go:66] "Starting communication layer"\nI0918 22:08:36.080849 1 simulator.go:210] "Start processing routine"\nI0918 22:08:36.081153 1 http_server_tls.go:59] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key"\nI0918 22:08:36.081212 1 grpc.go:129] "Server starting" protocol="gRPC" port=8000\nI0918 22:08:36.081854 1 http.go:117] "Server starting" protocol="HTTPS" port=8000\n' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=7) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 b'{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.964165174Z","logger":"setup","caller":"runner/runner.go:221","body":"GIE build","service.name":"llm-d-epp","commit-sha":"unknown","build-ref":"","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.964331384Z","logger":"setup","caller":"runner/runner.go:245","body":"Flags processed","service.name":"llm-d-epp","flags":{"allow-experimental-plugins":false,"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\\nkind: EndpointPickerConfig\\nplugins:\\n- type: disagg-headers-handler\\n- type: prefill-filter\\n- type: decode-filter\\n- type: always-disagg-pd-decider\\n- parameters:\\n deciders:\\n prefill: always-disagg-pd-decider\\n type: disagg-profile-handler\\n- type: prefix-cache-scorer\\n- type: queue-scorer\\n- type: max-score-picker\\n- parameters:\\n scheme: https\\n type: metrics-data-source\\nschedulingProfiles:\\n- name: prefill\\n plugins:\\n - pluginRef: prefill-filter\\n - pluginRef: queue-scorer\\n weight: 2\\n - pluginRef: prefix-cache-scorer\\n weight: 3\\n - pluginRef: max-score-picker\\n- name: decode\\n plugins:\\n - pluginRef: decode-filter\\n - pluginRef: queue-scorer\\n weight: 2\\n - pluginRef: prefix-cache-scorer\\n weight: 3\\n - pluginRef: max-score-picker\\n","disable-endpoint-subset-filter":false,"drain-timeout":30000000000,"emit-endpoint-scores":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"fairness-id-metric-label-limit":1000,"feature-gates":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-cert-dir":"","metrics-client-ca-file":"","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"plugin-state-staleness-threshold":300000000000,"pool-group":"inference.networking.k8s.io","pool-name":"scheduler-v06-pd-migration-test-inference-pool","pool-namespace":"e2e-test-llm-inference-service-9c386aa9","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tls-cipher-suites":{},"tls-min-version":"","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.964443574Z","logger":"setup.trace","caller":"tracing/telemetry.go:298","body":"init OTel trace exporter","service.name":"llm-d-epp","type":"otlp","protocol":"grpc","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.965509184Z","caller":"loader/configloader.go:99","body":"Loaded raw configuration","service.name":"llm-d-epp","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\\"deciders\\":{\\"prefill\\":\\"always-disagg-pd-decider\\"}}} {Type: prefix-cache-scorer} {Type: queue-scorer} {Type: max-score-picker} {Type: metrics-data-source, Parameters: {\\"scheme\\":\\"https\\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.965536494Z","logger":"setup","caller":"runner/runner.go:757","body":"Data layer: ENABLED","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.965801701Z","logger":"setup","caller":"runner/runner.go:362","body":"Raw config after phase one","service.name":"llm-d-epp","config":{"apiVersion":"llm-d.ai/v1alpha1", [e2e-llm-inference-service] "dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]},"severity_number":9}\n{"severity_text":"ERROR","timestamp":"2026-09-18T22:19:36.982725855Z","logger":"setup","caller":"runner/runner.go:387","body":"Failed to parse configuration","service.name":"llm-d-epp","error":"failed to load the configuration - configuration validation failed: plugin type \'disagg-headers-handler\' is not registered","severity_number":17,"stacktrace":"github.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).setup\\n\\t/workspace/cmd/epp/runner/runner.go:387\\ngithub.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).Run\\n\\t/workspace/cmd/epp/runner/runner.go:282\\nmain.run\\n\\t/workspace/cmd/epp/main.go:46\\nmain.main\\n\\t/workspace/cmd/epp/main.go:36\\nruntime.main\\n\\t/usr/lib/golang/src/runtime/proc.go:290"}\n' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (previous) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 b'{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.964165174Z","logger":"setup","caller":"runner/runner.go:221","body":"GIE build","service.name":"llm-d-epp","commit-sha":"unknown","build-ref":"","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.964331384Z","logger":"setup","caller":"runner/runner.go:245","body":"Flags processed","service.name":"llm-d-epp","flags":{"allow-experimental-plugins":false,"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\\nkind: EndpointPickerConfig\\nplugins:\\n- type: disagg-headers-handler\\n- type: prefill-filter\\n- type: decode-filter\\n- type: always-disagg-pd-decider\\n- parameters:\\n deciders:\\n prefill: always-disagg-pd-decider\\n type: disagg-profile-handler\\n- type: prefix-cache-scorer\\n- type: queue-scorer\\n- type: max-score-picker\\n- parameters:\\n scheme: https\\n type: metrics-data-source\\nschedulingProfiles:\\n- name: prefill\\n plugins:\\n - pluginRef: prefill-filter\\n - pluginRef: queue-scorer\\n weight: 2\\n - pluginRef: prefix-cache-scorer\\n weight: 3\\n - pluginRef: max-score-picker\\n- name: decode\\n plugins:\\n - pluginRef: decode-filter\\n - pluginRef: queue-scorer\\n weight: 2\\n - pluginRef: prefix-cache-scorer\\n weight: 3\\n - pluginRef: max-score-picker\\n","disable-endpoint-subset-filter":false,"drain-timeout":30000000000,"emit-endpoint-scores":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"fairness-id-metric-label-limit":1000,"feature-gates":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-cert-dir":"","metrics-client-ca-file":"","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"plugin-state-staleness-threshold":300000000000,"pool-group":"inference.networking.k8s.io","pool-name":"scheduler-v06-pd-migration-test-inference-pool","pool-namespace":"e2e-test-llm-inference-service-9c386aa9","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tls-cipher-suites":{},"tls-min-version":"","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.964443574Z","logger":"setup.trace","caller":"tracing/telemetry.go:298","body":"init OTel trace exporter","service.name":"llm-d-epp","type":"otlp","protocol":"grpc","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.965509184Z","caller":"loader/configloader.go:99","body":"Loaded raw configuration","service.name":"llm-d-epp","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\\"deciders\\":{\\"prefill\\":\\"always-disagg-pd-decider\\"}}} {Type: prefix-cache-scorer} {Type: queue-scorer} {Type: max-score-picker} {Type: metrics-data-source, Parameters: {\\"scheme\\":\\"https\\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.965536494Z","logger":"setup","caller":"runner/runner.go:757","body":"Data layer: ENABLED","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:19:36.965801701Z","logger":"setup","caller":"runner/runner.go:362","body":"Raw config after phase one","service.name":"llm-d-epp","config":{"apiVersion":"llm-d.ai/v1alpha1", [e2e-llm-inference-service] "dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]},"severity_number":9}\n{"severity_text":"ERROR","timestamp":"2026-09-18T22:19:36.982725855Z","logger":"setup","caller":"runner/runner.go:387","body":"Failed to parse configuration","service.name":"llm-d-epp","error":"failed to load the configuration - configuration validation failed: plugin type \'disagg-headers-handler\' is not registered","severity_number":17,"stacktrace":"github.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).setup\\n\\t/workspace/cmd/epp/runner/runner.go:387\\ngithub.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).Run\\n\\t/workspace/cmd/epp/runner/runner.go:282\\nmain.run\\n\\t/workspace/cmd/epp/main.go:46\\nmain.main\\n\\t/workspace/cmd/epp/main.go:36\\nruntime.main\\n\\t/usr/lib/golang/src/runtime/proc.go:290"}\n' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 5efd919a-1a6e-4773-9719-a6e23c42bf29 [e2e-llm-inference-service] resourceVersion: '25449' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:35Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - notReadyAddresses: [e2e-llm-inference-service] - ip: 10.134.0.34 [e2e-llm-inference-service] nodeName: ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp [e2e-llm-inference-service] uid: b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: de4d150e-1804-41f4-bbfd-f937a1a99320 [e2e-llm-inference-service] resourceVersion: '25662' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.44 [e2e-llm-inference-service] nodeName: ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9 [e2e-llm-inference-service] uid: e6c236b5-5165-46bf-a4b0-a9cbd13ca414 [e2e-llm-inference-service] - ip: 10.134.0.33 [e2e-llm-inference-service] nodeName: ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk [e2e-llm-inference-service] uid: 5d32db0e-866f-480a-85a4-0327717bc9af [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9 [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-5c479b44c4- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: e6c236b5-5165-46bf-a4b0-a9cbd13ca414 [e2e-llm-inference-service] resourceVersion: '25660' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c479b44c4 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.44/23"],"mac_address":"0a:58:0a:85:00:2c","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.44/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.44\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:2c\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c479b44c4 [e2e-llm-inference-service] uid: 17f2c5e9-4fd3-4b7d-b52c-69efb667c822 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-142-209 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17f2c5e9-4fd3-4b7d-b52c-69efb667c822"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.44"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-jzb5x [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-jzb5x [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000740000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 200ms [e2e-llm-inference-service] - --inter-token-latency [e2e-llm-inference-service] - 100ms [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-jzb5x [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000740000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] nodeName: ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c27,c19 [e2e-llm-inference-service] fsGroup: 1000740000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-d6hkm [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-bcctn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:36Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:36Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] hostIP: 10.0.142.209 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.142.209 [e2e-llm-inference-service] podIP: 10.133.0.44 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.44 [e2e-llm-inference-service] startTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-09-18T22:08:35Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar@sha256:8f710adc7cde2ba7289e68fdabd2c271737247727b0c8a9bdf00ad44bd09b34f [e2e-llm-inference-service] containerID: cri-o://54bb1fc56ff7a784d948b57f24ce57a599a7447a47b421328711de4f36642f8d [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-jzb5x [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000740000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000740000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-09-18T22:08:36Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim-dev@sha256:151063768d4a46d17a30ff512d74946b20cf653a2302601a6a823727fbbe4026 [e2e-llm-inference-service] containerID: cri-o://c54de8b367bb6d6e86e63ff4b129f89623387aefb2971c316cf4852fb53820fa [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-jzb5x [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000740000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000740000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-prefill-8665687897- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 5d32db0e-866f-480a-85a4-0327717bc9af [e2e-llm-inference-service] resourceVersion: '25644' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: '8665687897' [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.33/23"],"mac_address":"0a:58:0a:86:00:21","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.33/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.33\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:21\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-8665687897 [e2e-llm-inference-service] uid: 17cf4bc0-dbb1-4647-a36a-f9c5aa395e6e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-13 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17cf4bc0-dbb1-4647-a36a-f9c5aa395e6e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.33"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-bk2lw [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 5s [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-bk2lw [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000740000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c27,c19 [e2e-llm-inference-service] fsGroup: 1000740000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-d6hkm [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:36Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:44Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:44Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] hostIP: 10.0.141.13 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.13 [e2e-llm-inference-service] podIP: 10.134.0.33 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.33 [e2e-llm-inference-service] startTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-09-18T22:08:36Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim-dev@sha256:151063768d4a46d17a30ff512d74946b20cf653a2302601a6a823727fbbe4026 [e2e-llm-inference-service] containerID: cri-o://87f4cc7939fe9d118b854ecd7a84c1fc1648d25890a485a6080388341937fa51 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-bk2lw [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000740000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000740000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa [e2e-llm-inference-service] resourceVersion: '32390' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6db4b76ccf [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.34/23"],"mac_address":"0a:58:0a:86:00:22","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.34/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.34\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:22\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf [e2e-llm-inference-service] uid: 4b9b609e-0436-41f8-adb8-5dc4d6b76e7c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-13 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4b9b609e-0436-41f8-adb8-5dc4d6b76e7c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:19:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.34"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-2s7vp [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- type: prefix-cache-scorer\n- type: queue-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\n- parameters:\n scheme: https\n type: metrics-data-source\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-2s7vp [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000740000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c27,c19 [e2e-llm-inference-service] fsGroup: 1000740000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-d6hkm [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-grjtq [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:35Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] hostIP: 10.0.141.13 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.13 [e2e-llm-inference-service] podIP: 10.134.0.34 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.34 [e2e-llm-inference-service] startTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: CrashLoopBackOff [e2e-llm-inference-service] message: back-off 5m0s restarting failed container=main pod=scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp_e2e-test-llm-inference-service-9c386aa9(b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa) [e2e-llm-inference-service] lastState: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 1 [e2e-llm-inference-service] reason: Error [e2e-llm-inference-service] message: 'e one","service.name":"llm-d-epp","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]},"severity_number":9} [e2e-llm-inference-service] [e2e-llm-inference-service] {"severity_text":"ERROR","timestamp":"2026-09-18T22:19:36.982725855Z","logger":"setup","caller":"runner/runner.go:387","body":"Failed [e2e-llm-inference-service] to parse configuration","service.name":"llm-d-epp","error":"failed to load [e2e-llm-inference-service] the configuration - configuration validation failed: plugin type ''disagg-headers-handler'' [e2e-llm-inference-service] is not registered","severity_number":17,"stacktrace":"github.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).setup\n\t/workspace/cmd/epp/runner/runner.go:387\ngithub.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).Run\n\t/workspace/cmd/epp/runner/runner.go:282\nmain.run\n\t/workspace/cmd/epp/main.go:46\nmain.main\n\t/workspace/cmd/epp/main.go:36\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:290"} [e2e-llm-inference-service] [e2e-llm-inference-service] ' [e2e-llm-inference-service] startedAt: '2026-09-18T22:19:36Z' [e2e-llm-inference-service] finishedAt: '2026-09-18T22:19:36Z' [e2e-llm-inference-service] containerID: cri-o://dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6 [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 7 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:c12f30f05219c4138421584f2781f32dda09373ef168b71f854861f529c0792e [e2e-llm-inference-service] containerID: cri-o://dc90c5ce045167735f501f43986d17087d9d6ee8effc09eed7d057d45d9ee3f6 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-2s7vp [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000740000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000740000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 43a8b6a1-0b9e-4fec-a087-6435d6393c8a [e2e-llm-inference-service] resourceVersion: '25317' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: scheduler-v06-pd-migration-test-epp-sa-dockercfg-grjtq [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"scheduler-v06-pd-migration-test-epp-sa-dockercfg-grjtq"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-d6hkm"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-d6hkm [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-grjtq [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-d6hkm [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-grjtq [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: fbc56038-7955-45db-a045-0f9a8b1dda51 [e2e-llm-inference-service] resourceVersion: '25273' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: scheduler-v06-pd-migration-test-kserve-dockercfg-bcctn [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"scheduler-v06-pd-migration-test-kserve-dockercfg-bcctn"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-d6hkm"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-d6hkm [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-bcctn [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-d6hkm [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-bcctn [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 2d20cf83-b924-454e-9e2b-720a24dd8d6f [e2e-llm-inference-service] resourceVersion: '25340' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.114.88 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.114.88 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 75410a6f-5fa8-4302-934a-a57cafcda7fc [e2e-llm-inference-service] resourceVersion: '25307' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.80.91 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.80.91 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 5cd33002-80a6-49a1-a040-aa410d876dbd [e2e-llm-inference-service] resourceVersion: '25664' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 200ms [e2e-llm-inference-service] - --inter-token-latency [e2e-llm-inference-service] - 100ms [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-5c479b44c4" has successfully [e2e-llm-inference-service] progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: c01de8e7-212b-4d43-839a-7fe1471bff25 [e2e-llm-inference-service] resourceVersion: '25649' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 5s [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:08:44Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:44Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:08:44Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-prefill-8665687897" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: d2adfcc5-043c-4dc8-b6c8-b4e534116540 [e2e-llm-inference-service] resourceVersion: '31682' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:18:34Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: max-score-picker\n- parameters:\n scheme:\ [e2e-llm-inference-service] \ https\n type: metrics-data-source\nschedulingProfiles:\n- name: prefill\n\ [e2e-llm-inference-service] \ plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:18:34Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:18:34Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c479b44c4 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 17f2c5e9-4fd3-4b7d-b52c-69efb667c822 [e2e-llm-inference-service] resourceVersion: '25663' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c479b44c4 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] uid: 5cd33002-80a6-49a1-a040-aa410d876dbd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5cd33002-80a6-49a1-a040-aa410d876dbd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c479b44c4 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c479b44c4 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 200ms [e2e-llm-inference-service] - --inter-token-latency [e2e-llm-inference-service] - 100ms [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-8665687897 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 17cf4bc0-dbb1-4647-a36a-f9c5aa395e6e [e2e-llm-inference-service] resourceVersion: '25647' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: '8665687897' [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] uid: c01de8e7-212b-4d43-839a-7fe1471bff25 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c01de8e7-212b-4d43-839a-7fe1471bff25"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: '8665687897' [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: '8665687897' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 5s [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 4b9b609e-0436-41f8-adb8-5dc4d6b76e7c [e2e-llm-inference-service] resourceVersion: '25356' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6db4b76ccf [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler [e2e-llm-inference-service] uid: d2adfcc5-043c-4dc8-b6c8-b4e534116540 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d2adfcc5-043c-4dc8-b6c8-b4e534116540"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6db4b76ccf [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6db4b76ccf [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: max-score-picker\n- parameters:\n scheme:\ [e2e-llm-inference-service] \ https\n type: metrics-data-source\nschedulingProfiles:\n- name: prefill\n\ [e2e-llm-inference-service] \ plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-prometheus-scraping [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: c28f065a-ea26-4e25-9423-700862fd073d [e2e-llm-inference-service] resourceVersion: '25509' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llm-monitoring [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: networking.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:ingress: {} [e2e-llm-inference-service] f:podSelector: {} [e2e-llm-inference-service] f:policyTypes: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] podSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ingress: [e2e-llm-inference-service] - ports: [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] from: [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: openshift-monitoring [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: openshift-user-workload-monitoring [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: redhat-ods-monitoring [e2e-llm-inference-service] - ports: [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] from: [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: openshift-ingress [e2e-llm-inference-service] - from: [e2e-llm-inference-service] - podSelector: {} [e2e-llm-inference-service] policyTypes: [e2e-llm-inference-service] - Ingress [e2e-llm-inference-service] apiVersion: networking.k8s.io/v1 [e2e-llm-inference-service] kind: NetworkPolicy [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 6dbf4274-87a2-45f5-83ae-e9d7f62ea9ff [e2e-llm-inference-service] resourceVersion: '25334' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 807335a8-45e5-4513-b230-85395f38c722 [e2e-llm-inference-service] resourceVersion: '25283' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: ff75a206-8579-4975-a55e-0426c21bc116 [e2e-llm-inference-service] resourceVersion: '25330' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - patch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - events [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: e99c44cc-d1a5-4c5f-8e59-de5f76cc8626 [e2e-llm-inference-service] resourceVersion: '25279' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service-q4hgh [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 232c6b47-09e6-4b66-b3be-695ff7c1f0f0 [e2e-llm-inference-service] resourceVersion: '25448' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] uid: 2d20cf83-b924-454e-9e2b-720a24dd8d6f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:35Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2d20cf83-b924-454e-9e2b-720a24dd8d6f"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.34 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] serving: false [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-6dc8nqp [e2e-llm-inference-service] uid: b4cc8ea0-9ab3-4a08-8ce0-8a5d663ab9fa [e2e-llm-inference-service] nodeName: ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc-lqptq [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 0ca6c367-ce41-459f-8182-a287a4435c49 [e2e-llm-inference-service] resourceVersion: '25661' [e2e-llm-inference-service] generation: 5 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] uid: 75410a6f-5fa8-4302-934a-a57cafcda7fc [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"75410a6f-5fa8-4302-934a-a57cafcda7fc"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.33 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk [e2e-llm-inference-service] uid: 5d32db0e-866f-480a-85a4-0327717bc9af [e2e-llm-inference-service] nodeName: ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.44 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9 [e2e-llm-inference-service] uid: e6c236b5-5165-46bf-a4b0-a9cbd13ca414 [e2e-llm-inference-service] nodeName: ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 6dbf4274-87a2-45f5-83ae-e9d7f62ea9ff [e2e-llm-inference-service] resourceVersion: '25334' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-9c386aa9:scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: 807335a8-45e5-4513-b230-85395f38c722 [e2e-llm-inference-service] resourceVersion: '25283' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-9c386aa9:scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: ff75a206-8579-4975-a55e-0426c21bc116 [e2e-llm-inference-service] resourceVersion: '25330' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - patch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - events [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] uid: e99c44cc-d1a5-4c5f-8e59-de5f76cc8626 [e2e-llm-inference-service] resourceVersion: '25279' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25688' [e2e-llm-inference-service] uid: d73e737f-002a-468e-8ecf-542744deca9a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25688' [e2e-llm-inference-service] uid: d73e737f-002a-468e-8ecf-542744deca9a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-9c386aa9/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25672' [e2e-llm-inference-service] uid: dfe1d4f2-71e4-44b9-8777-e2b2213adbd8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:45Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:36Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:36Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:08:37Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25563' [e2e-llm-inference-service] uid: 78b47246-a404-4b15-984e-792b8af83f2d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:36Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:08:37Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25414' [e2e-llm-inference-service] uid: badfe994-0436-456e-bfb3-734b02b32b83 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25690' [e2e-llm-inference-service] uid: f8d62346-990b-46f9-8c8c-cebf81def5c8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25439' [e2e-llm-inference-service] uid: e409e779-e2cb-401c-9bd2-b29e92b2227a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25414' [e2e-llm-inference-service] uid: badfe994-0436-456e-bfb3-734b02b32b83 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25690' [e2e-llm-inference-service] uid: f8d62346-990b-46f9-8c8c-cebf81def5c8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25439' [e2e-llm-inference-service] uid: e409e779-e2cb-401c-9bd2-b29e92b2227a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25414' [e2e-llm-inference-service] uid: badfe994-0436-456e-bfb3-734b02b32b83 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:46Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25690' [e2e-llm-inference-service] uid: f8d62346-990b-46f9-8c8c-cebf81def5c8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:34Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25439' [e2e-llm-inference-service] uid: e409e779-e2cb-401c-9bd2-b29e92b2227a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-9c386aa9.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8bccd30e-0934-46f0-b22c-8d1b18f270f4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:08:33Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 8bccd30e-0934-46f0-b22c-8d1b18f270f4 [e2e-llm-inference-service] resourceVersion: '25349' [e2e-llm-inference-service] uid: 258cbbd3-4544-4edd-ae19-90c26bf927f1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c479b44c4 [e2e-llm-inference-service] timestamp: '2026-09-18T22:23:17Z' [e2e-llm-inference-service] window: 14.972s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 3513358n [e2e-llm-inference-service] memory: 21304Ki [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 11131790n [e2e-llm-inference-service] memory: 23988Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-8665687897-22jlk [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-9c386aa9 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: '8665687897' [e2e-llm-inference-service] timestamp: '2026-09-18T22:23:23Z' [e2e-llm-inference-service] window: 13.705s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 11242028n [e2e-llm-inference-service] memory: 24080Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-09-18T22:23:30.445734] end - ❌ 902.518s: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:08:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:190 Skipping deletion of namespace e2e-test-llm-inference-service-9c386aa9 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-v06-nonzero-threshold-migration', 'workload-llmd-simulator-pd'], prom... {'name': 'workload-llmd-simulator-pd-sche-132dd369'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-oci", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Same migration path, but with the legacy plugin config supplied via [e2e-llm-inference-service] # spec.router.scheduler.template.containers[].args instead of [e2e-llm-inference-service] # spec.router.scheduler.config.inline. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-template-args-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-template-args-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:922: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...shold-f63f7c1e'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-132dd369'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {} [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....o-threshold-f63f7c1e'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-132dd369'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1432: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f9a5eeda700> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] ^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1443: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:24:23Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1427: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-40b85f95 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig router-managed-scheduler-v06-th-149ecf0d in namespace e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig router-managed-scheduler-v06-th-149ecf0d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig router-managed-scheduler-v06-th-149ecf0d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig scheduler-v06-nonzero-threshold-f63f7c1e in namespace e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig scheduler-v06-nonzero-threshold-f63f7c1e [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig scheduler-v06-nonzero-threshold-f63f7c1e [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-132dd369 in namespace e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-132dd369 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-132dd369 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-09-18T22:23:31.195280] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'scheduler-v06-nonzero-threshold-migration', 'workload-llmd-simulator-pd'], prompt='KServe is a', service_name='scheduler-v06-threshold-migration-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-40b85f95', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-threshold-migration-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-40b85f95', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-th-149ecf0d'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-nonzero-threshold-f63f7c1e'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-132dd369'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-09-18T22:23:31.208163] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-threshold-migration-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-40b85f95', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-th-149ecf0d'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-nonzero-threshold-f63f7c1e'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-132dd369'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-09-18T22:23:31.356000] end - ✅ in 0.148s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-09-18T22:23:31.356089] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-threshold-migration-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-40b85f95', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-th-149ecf0d'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-nonzero-threshold-f63f7c1e'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-132dd369'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:23:53Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-route]', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'message': 'Inference Pool e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-route]', 'reason': 'WaitingForGateway', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-route]', 'reason': 'WaitingForGateway', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:24:19Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:24:23Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:24:23Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:24:23Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:24:23Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1447 Timed out waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:24:23Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-09-18T22:38:31.425756] end - ❌ 900.069s: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:24:23Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:959 [router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] ❌ ERROR: Failed to call llm inference service scheduler-v06-threshold-migration-test: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:24:23Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:274 # Diagnostics for 'scheduler-v06-threshold-migration-test' in 'e2e-test-llm-inference-service-40b85f95' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:275 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:278 # LLMInferenceService scheduler-v06-threshold-migration-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:287 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:31Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:23:31Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:23:37Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-eppconfig-default: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-eppconfig-default-pd: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-sglang-template: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:33:52Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] resourceVersion: '44233' [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-scheduler-v06-th-149ecf0d [e2e-llm-inference-service] - name: scheduler-v06-nonzero-threshold-f63f7c1e [e2e-llm-inference-service] - name: workload-llmd-simulator-pd-sche-132dd369 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-eppconfig-default: kserve-config-llm-scheduler-eppconfig-default [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-eppconfig-default-pd: kserve-config-llm-scheduler-eppconfig-default-pd [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-sglang-template: kserve-config-sglang-template [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:24:23Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:24:19Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:24:19Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:24:19Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:23:53Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:33:52Z' [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:33:52Z' [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:33:52Z' [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:24:19Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-40b85f95 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7 to ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:54 e2e-test-llm-inference-service-40b85f95 multus Normal AddedInterface Add eth0 [10.133.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:54 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-142-209.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:54 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-142-209.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:54 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-142-209.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:54 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-142-209.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:55 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-142-209.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:55 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-142-209.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-40b85f95 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2 to ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 multus Normal AddedInterface Add eth0 [10.132.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-130-79.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-130-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-130-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-prefill-64446cf4ff from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-40b85f95 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr to ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 multus Normal AddedInterface Add eth0 [10.132.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:35:00 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-130-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:35:00 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-130-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:35:00 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-130-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:33:56 e2e-test-llm-inference-service-40b85f95 kubelet/ip-10-0-130-79.ec2.internal Warning BackOff Back-off restarting failed container main in pod scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-59b84dfb55 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:41 e2e-test-llm-inference-service-40b85f95 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-threshold-migration-test-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-threshold-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:50 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:50 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:50 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:23:51 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-e884f9b5c5d28bdfbee5263abf56adc6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:24:22 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:24:17 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-09-18 22:24:18 e2e-test-llm-inference-service-40b85f95 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:252 ### EndpointSlice scheduler-v06-threshold-migration-test-epp-service-jf9sd (service=scheduler-v06-threshold-migration-test-epp-service): [{'addresses': ['10.132.0.35'], 'ready': False, 'targetRef': 'scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:252 ### EndpointSlice scheduler-v06-threshold-migration-test-inference-po-ip-32dhsggd (service=scheduler-v06-threshold-migration-test-inference-po-ip-32d3e584): [{'addresses': ['10.133.0.49'], 'ready': True, 'targetRef': 'scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7'}, {'addresses': ['10.132.0.34'], 'ready': True, 'targetRef': 'scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:252 ### EndpointSlice scheduler-v06-threshold-migration-test-kserve-workload-svc47fb2 (service=scheduler-v06-threshold-migration-test-kserve-workload-svc): [{'addresses': ['10.132.0.34'], 'ready': True, 'targetRef': 'scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2'}, {'addresses': ['10.133.0.49'], 'ready': True, 'targetRef': 'scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 b'{"severity_text":"INFO","timestamp":"2026-09-18T22:23:54.352209095Z","body":"Proxy starting","service.name":"llm-d-router-disagg-sidecar","Built on":"","From Git SHA":"unknown","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:54.352270316Z","body":"Proxy configuration","service.name":"llm-d-router-disagg-sidecar","config":"{\\"Port\\":\\"8000\\",\\"KVConnector\\":\\"nixlv2\\",\\"ECConnector\\":\\"\\",\\"DataParallelSize\\":1,\\"MaxIdleConnsPerHost\\":1024,\\"EnablePrefillerSampling\\":false,\\"PrefillMaxRetries\\":0,\\"PrefillRetryBackoff\\":200000000,\\"UseTLSForPrefiller\\":true,\\"UseTLSForDecoder\\":true,\\"UseTLSForEncoder\\":false,\\"InsecureSkipVerifyForPrefiller\\":false,\\"InsecureSkipVerifyForEncoder\\":false,\\"InsecureSkipVerifyForDecoder\\":false,\\"SecureServing\\":true,\\"CertPath\\":\\"/var/run/kserve/tls\\",\\"TLSMinVersion\\":0,\\"TLSCipherSuites\\":null,\\"MetricsPort\\":0,\\"MetricsCertDir\\":\\"\\",\\"MooncakeBootstrapPort\\":8998,\\"P2PConnectorPort\\":7777,\\"EnableP2PPull\\":false,\\"EnableSSRFProtection\\":true,\\"InferencePoolNamespace\\":\\"e2e-test-llm-inference-service-40b85f95\\",\\"InferencePoolName\\":\\"scheduler-v06-threshold-migration-test-inference-pool\\",\\"PoolGroup\\":\\"inference.networking.x-k8s.io\\",\\"DecodeChunkSize\\":0,\\"Tracing\\":false,\\"MoRIIOWriteMode\\":false,\\"MoRIIODecodeNotifyPort\\":61005,\\"MoRIIODecodeHandshakePort\\":6301,\\"MoRIIODecodePodIP\\":\\"\\",\\"MoRIIOParallelDispatch\\":false,\\"MoRIIOParallelDecodeWaitTimeout\\":30000000000,\\"MoRIIOPrefillHandshakePort\\":6301,\\"MoRIIOPrefillNotifyPort\\":61005,\\"MoRIIOTPSize\\":1,\\"MoRIIODPSize\\":1,\\"MoRIIORemoteHosts\\":null,\\"MoRIIODPSizeLocal\\":0,\\"MoRIIODecodeHosts\\":null,\\"MoRIIORemoteHostSpecs\\":null,\\"MoRIIODecodeHostSpecs\\":null,\\"MoRIIODecodePodIPSpec\\":\\"\\",\\"DecoderURL\\":\\"https://localhost:8001\\"}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:54.353740143Z","logger":"allowlist-validator","body":"starting SSRF protection allowlist validator","service.name":"llm-d-router-disagg-sidecar","namespace":"e2e-test-llm-inference-service-40b85f95","poolName":"scheduler-v06-threshold-migration-test-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:54.366806822Z","logger":"allowlist-validator","body":"InferencePool added","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:54.373797617Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":2,"targets":{"10.132.0.34":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:54.373902932Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":2,"targets":{"10.132.0.34":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:54.454805145Z","logger":"allowlist-validator","body":"allowlist validator started successfully","service.name":"llm-d-router-disagg-sidecar","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:54.456240813Z","logger":"proxy server on port 8000","body":"server TLS configured","service.name":"llm-d-router-disagg-sidecar","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:54.456265719Z","logger":"proxy server on port 8000","body":"starting","service.name":"llm-d-router-disagg-sidecar","addr":"[::]:8000","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:54.997599754Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132 [e2e-llm-inference-service] .0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:23:56.008336031Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:24:01.420494146Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:24:01.498989929Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:24:01.640369834Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:24:02.054727486Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:24:13.063940257Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:24:24.367290541Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:24:24.376817725Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:24:24.376914865Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:24:54.368013172Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:24:54.376603264Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler [e2e-llm-inference-service] -v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:24:54.376670208Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:25:24.369307322Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:25:24.37781659Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:25:24.377875559Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:25:54.369581317Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:25:54.378343886Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:25:54.378406387Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:26:24.37056804Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:26:24.379005498Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:26:24.379066604Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:26:54.370665798Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:26:54.379105266Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-rout [e2e-llm-inference-service] er-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:26:54.379156223Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:27:24.37103952Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:27:24.380704023Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:27:24.380765016Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:27:54.371895678Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:27:54.379686592Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:27:54.37974045Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:28:24.372086019Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:28:24.380480079Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:28:24.3805472Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:28:54.372932823Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:28:54.38107545 [e2e-llm-inference-service] 1Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:28:54.381130776Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:29:24.374120673Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:29:24.382485606Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:29:24.382539333Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:29:54.374888344Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:29:54.383166741Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:29:54.383232329Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:30:24.375667663Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:30:24.384739403Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:30:24.384791992Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:30:54.376124524Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-p [e2e-llm-inference-service] ool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:30:54.38423422Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:30:54.384295998Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:31:24.376257745Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:31:24.385796337Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:31:24.385879128Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:31:54.377012252Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:31:54.386032651Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:31:54.386083285Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:32:24.378294579Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:32:24.38691506Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:32:24.386990737Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:32:54.379546299Z","logger":"allowlist-validator","body":"InferencePool updated","service.name [e2e-llm-inference-service] ":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:32:54.390733479Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:32:54.390797652Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:33:24.38061941Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:33:24.392817611Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:33:24.392898272Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:33:54.380765834Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:33:54.389308444Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:33:54.389365713Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:34:24.38133841Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:34:24.390296252Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:34:24.390348422Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:34: [e2e-llm-inference-service] 54.381448285Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:34:54.39007856Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:34:54.390149314Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:24.382073169Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:24.391746148Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:24.391814951Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:54.383130815Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:54.391939204Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:54.39200731Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:36:24.384147383Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:36:24.392457933Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:36:24.392518418Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-6 [e2e-llm-inference-service] 4444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:36:54.385399446Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:36:54.394326337Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:36:54.394388479Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:37:24.3861332Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:37:24.39489017Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:37:24.39493891Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:37:54.386202973Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:37:54.394388187Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:37:54.394448665Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:38:24.386301954Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"scheduler-v06-threshold-migration-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:38:24.396598119Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:38:24.39686012Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.34":{},"10.133.0.49":{},"scheduler-v06-threshold-migration-te [e2e-llm-inference-service] st-kserve-59b84dfb55-qpnd7":{},"scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2":{}},"severity_number":9}\n' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 b'I0918 22:23:55.061766 1 config.go:808] "Configuration:" =<\n\t{\n\t "DisplayModelName": "facebook/opt-125m",\n\t "IP": "",\n\t "PodName": "",\n\t "PodNameSpace": "",\n\t "VllmDevMode": false,\n\t "block-size": 16,\n\t "data-parallel-rank": -1,\n\t "data-parallel-size": 1,\n\t "dataset-in-memory": false,\n\t "dataset-path": "",\n\t "dataset-table-name": "llmd",\n\t "dataset-url": "",\n\t "default-embedding-dimensions": 384,\n\t "ec-transfer-config": "",\n\t "enable-kvcache": false,\n\t "enable-prefix-caching": false,\n\t "enable-request-id-headers": false,\n\t "enable-sleep-mode": false,\n\t "enforce-eager": false,\n\t "event-batch-size": 16,\n\t "failure-injection-rate": 0,\n\t "failure-types": null,\n\t "fake-metrics": null,\n\t "fake-metrics-refresh-interval": "100ms",\n\t "force-dummy-tokenizer": true,\n\t "global-cache-hit-threshold": 0,\n\t "hash-seed": "",\n\t "image-emission-rate": 0,\n\t "inter-token-latency": "100ms",\n\t "inter-token-latency-std-dev": "0s",\n\t "kv-cache-size": 1024,\n\t "kv-cache-transfer-latency": "0s",\n\t "kv-cache-transfer-latency-std-dev": "0s",\n\t "kv-cache-transfer-time-per-token": "0s",\n\t "kv-cache-transfer-time-std-dev": "0s",\n\t "latency-calculator": "",\n\t "log-http": false,\n\t "lora-modules": null,\n\t "max-cpu-loras": 1,\n\t "max-loras": 1,\n\t "max-model-len": 1024,\n\t "max-num-seqs": 5,\n\t "max-request-body-size-mb": 4,\n\t "max-tool-call-array-param-length": 5,\n\t "max-tool-call-integer-param": 100,\n\t "max-tool-call-number-param": 100,\n\t "max-waiting-queue-length": 1000,\n\t "min-tool-call-array-param-length": 1,\n\t "min-tool-call-integer-param": 0,\n\t "min-tool-call-number-param": 0,\n\t "mm-encoder-only": false,\n\t "mm-processor-kwargs": "",\n\t "mm-render-timeout": "1m0s",\n\t "mode": "random",\n\t "model": "facebook/opt-125m",\n\t "object-tool-call-not-required-field-probability": 50,\n\t "omni": false,\n\t "port": 8001,\n\t "prefill-overhead": "0s",\n\t "prefill-time-per-token": "0s",\n\t "prefill-time-std-dev": "0s",\n\t "render-timeout": "30s",\n\t "render-url": "http://localhost:8082",\n\t "seed": 1789770235061278200,\n\t "self-signed-certs": false,\n\t "served-model-name": [\n\t "facebook/opt-125m"\n\t ],\n\t "ssl-certfile": "/var/run/kserve/tls/tls.crt",\n\t "ssl-keyfile": "/var/run/kserve/tls/tls.key",\n\t "startup-duration": "0s",\n\t "time-factor-under-load": 1,\n\t "time-to-first-token": "200ms",\n\t "time-to-first-token-std-dev": "0s",\n\t "time-to-generate-image": "0s",\n\t "time-to-generate-image-std-dev": "0s",\n\t "tool-call-not-required-param-probability": 50,\n\t "use-vllm-map-event-format": false,\n\t "zmq-endpoint": "tcp://127.0.0.1:5557"\n\t}\n >\nI0918 22:23:55.061829 1 tokenizer.go:60] "Force dummy tokenizer flag is set, using simulated tokenizer" model="facebook/opt-125m"\nI0918 22:23:55.067045 1 context.go:216] "No dataset path or URL provided, using random text for responses"\nI0918 22:23:55.067107 1 communication.go:66] "Starting communication layer"\nI0918 22:23:55.067211 1 simulator.go:210] "Start processing routine"\nI0918 22:23:55.067443 1 http_server_tls.go:59] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key"\nI0918 22:23:55.067497 1 grpc.go:129] "Server starting" protocol="gRPC" port=8001\nI0918 22:23:55.068223 1 http.go:117] "Server starting" protocol="HTTPS" port=8001\n' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 b'I0918 22:23:51.764894 1 config.go:808] "Configuration:" =<\n\t{\n\t "DisplayModelName": "facebook/opt-125m",\n\t "IP": "",\n\t "PodName": "",\n\t "PodNameSpace": "",\n\t "VllmDevMode": false,\n\t "block-size": 16,\n\t "data-parallel-rank": -1,\n\t "data-parallel-size": 1,\n\t "dataset-in-memory": false,\n\t "dataset-path": "",\n\t "dataset-table-name": "llmd",\n\t "dataset-url": "",\n\t "default-embedding-dimensions": 384,\n\t "ec-transfer-config": "",\n\t "enable-kvcache": false,\n\t "enable-prefix-caching": false,\n\t "enable-request-id-headers": false,\n\t "enable-sleep-mode": false,\n\t "enforce-eager": false,\n\t "event-batch-size": 16,\n\t "failure-injection-rate": 0,\n\t "failure-types": null,\n\t "fake-metrics": null,\n\t "fake-metrics-refresh-interval": "100ms",\n\t "force-dummy-tokenizer": true,\n\t "global-cache-hit-threshold": 0,\n\t "hash-seed": "",\n\t "image-emission-rate": 0,\n\t "inter-token-latency": "0s",\n\t "inter-token-latency-std-dev": "0s",\n\t "kv-cache-size": 1024,\n\t "kv-cache-transfer-latency": "0s",\n\t "kv-cache-transfer-latency-std-dev": "0s",\n\t "kv-cache-transfer-time-per-token": "0s",\n\t "kv-cache-transfer-time-std-dev": "0s",\n\t "latency-calculator": "",\n\t "log-http": false,\n\t "lora-modules": null,\n\t "max-cpu-loras": 1,\n\t "max-loras": 1,\n\t "max-model-len": 1024,\n\t "max-num-seqs": 5,\n\t "max-request-body-size-mb": 4,\n\t "max-tool-call-array-param-length": 5,\n\t "max-tool-call-integer-param": 100,\n\t "max-tool-call-number-param": 100,\n\t "max-waiting-queue-length": 1000,\n\t "min-tool-call-array-param-length": 1,\n\t "min-tool-call-integer-param": 0,\n\t "min-tool-call-number-param": 0,\n\t "mm-encoder-only": false,\n\t "mm-processor-kwargs": "",\n\t "mm-render-timeout": "1m0s",\n\t "mode": "random",\n\t "model": "facebook/opt-125m",\n\t "object-tool-call-not-required-field-probability": 50,\n\t "omni": false,\n\t "port": 8000,\n\t "prefill-overhead": "0s",\n\t "prefill-time-per-token": "0s",\n\t "prefill-time-std-dev": "0s",\n\t "render-timeout": "30s",\n\t "render-url": "http://localhost:8082",\n\t "seed": 1789770231764370000,\n\t "self-signed-certs": false,\n\t "served-model-name": [\n\t "facebook/opt-125m"\n\t ],\n\t "ssl-certfile": "/var/run/kserve/tls/tls.crt",\n\t "ssl-keyfile": "/var/run/kserve/tls/tls.key",\n\t "startup-duration": "0s",\n\t "time-factor-under-load": 1,\n\t "time-to-first-token": "5s",\n\t "time-to-first-token-std-dev": "0s",\n\t "time-to-generate-image": "0s",\n\t "time-to-generate-image-std-dev": "0s",\n\t "tool-call-not-required-param-probability": 50,\n\t "use-vllm-map-event-format": false,\n\t "zmq-endpoint": "tcp://127.0.0.1:5557"\n\t}\n >\nI0918 22:23:51.764972 1 tokenizer.go:60] "Force dummy tokenizer flag is set, using simulated tokenizer" model="facebook/opt-125m"\nI0918 22:23:51.770674 1 context.go:216] "No dataset path or URL provided, using random text for responses"\nI0918 22:23:51.770898 1 communication.go:66] "Starting communication layer"\nI0918 22:23:51.771404 1 simulator.go:210] "Start processing routine"\nI0918 22:23:51.771532 1 http_server_tls.go:59] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key"\nI0918 22:23:51.771614 1 grpc.go:129] "Server starting" protocol="gRPC" port=8000\nI0918 22:23:51.772250 1 http.go:117] "Server starting" protocol="HTTPS" port=8000\n' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=7) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 b'{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.698358986Z","logger":"setup","caller":"runner/runner.go:221","body":"GIE build","service.name":"llm-d-epp","commit-sha":"unknown","build-ref":"","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.698462553Z","logger":"setup","caller":"runner/runner.go:245","body":"Flags processed","service.name":"llm-d-epp","flags":{"allow-experimental-plugins":false,"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\\nkind: EndpointPickerConfig\\nplugins:\\n- type: disagg-headers-handler\\n- type: prefill-filter\\n- type: decode-filter\\n- parameters:\\n nonCachedTokens: 25\\n type: prefix-based-pd-decider\\n- parameters:\\n deciders:\\n prefill: prefix-based-pd-decider\\n type: disagg-profile-handler\\n- type: prefix-cache-scorer\\n- type: queue-scorer\\n- type: max-score-picker\\n- parameters:\\n scheme: https\\n type: metrics-data-source\\nschedulingProfiles:\\n- name: prefill\\n plugins:\\n - pluginRef: prefill-filter\\n - pluginRef: queue-scorer\\n weight: 2\\n - pluginRef: prefix-cache-scorer\\n weight: 3\\n - pluginRef: max-score-picker\\n- name: decode\\n plugins:\\n - pluginRef: decode-filter\\n - pluginRef: queue-scorer\\n weight: 2\\n - pluginRef: prefix-cache-scorer\\n weight: 3\\n - pluginRef: max-score-picker\\n","disable-endpoint-subset-filter":false,"drain-timeout":30000000000,"emit-endpoint-scores":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"fairness-id-metric-label-limit":1000,"feature-gates":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-cert-dir":"","metrics-client-ca-file":"","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"plugin-state-staleness-threshold":300000000000,"pool-group":"inference.networking.k8s.io","pool-name":"scheduler-v06-threshold-migration-test-inference-pool","pool-namespace":"e2e-test-llm-inference-service-40b85f95","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tls-cipher-suites":{},"tls-min-version":"","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.698573686Z","logger":"setup.trace","caller":"tracing/telemetry.go:298","body":"init OTel trace exporter","service.name":"llm-d-epp","type":"otlp","protocol":"grpc","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.699255717Z","caller":"loader/configloader.go:99","body":"Loaded raw configuration","service.name":"llm-d-epp","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: prefix-based-pd-decider, Parameters: {\\"nonCachedTokens\\":25}} {Type: disagg-profile-handler, Parameters: {\\"deciders\\":{\\"prefill\\":\\"prefix-based-pd-decider\\"}}} {Type: prefix-cache-scorer} {Type: queue-scorer} {Type: max-score-picker} {Type: metrics-data-source, Parameters: {\\"scheme\\":\\"https\\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.699309851Z","logger":"setup","caller":"runner/runner.go:757","body":"Data layer: ENABLED","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.699599704Z","logger":"setup","caller":"runner/runner.go:362","body":"Raw config af [e2e-llm-inference-service] ter phase one","service.name":"llm-d-epp","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"prefix-based-pd-decider","parameters":{"nonCachedTokens":25},"type":"prefix-based-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"prefix-based-pd-decider"}},"type":"disagg-profile-handler"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]},"severity_number":9}\n{"severity_text":"ERROR","timestamp":"2026-09-18T22:35:00.718035952Z","logger":"setup","caller":"runner/runner.go:387","body":"Failed to parse configuration","service.name":"llm-d-epp","error":"failed to load the configuration - configuration validation failed: plugin type \'disagg-headers-handler\' is not registered","severity_number":17,"stacktrace":"github.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).setup\\n\\t/workspace/cmd/epp/runner/runner.go:387\\ngithub.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).Run\\n\\t/workspace/cmd/epp/runner/runner.go:282\\nmain.run\\n\\t/workspace/cmd/epp/main.go:46\\nmain.main\\n\\t/workspace/cmd/epp/main.go:36\\nruntime.main\\n\\t/usr/lib/golang/src/runtime/proc.go:290"}\n' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (previous) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 b'{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.698358986Z","logger":"setup","caller":"runner/runner.go:221","body":"GIE build","service.name":"llm-d-epp","commit-sha":"unknown","build-ref":"","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.698462553Z","logger":"setup","caller":"runner/runner.go:245","body":"Flags processed","service.name":"llm-d-epp","flags":{"allow-experimental-plugins":false,"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\\nkind: EndpointPickerConfig\\nplugins:\\n- type: disagg-headers-handler\\n- type: prefill-filter\\n- type: decode-filter\\n- parameters:\\n nonCachedTokens: 25\\n type: prefix-based-pd-decider\\n- parameters:\\n deciders:\\n prefill: prefix-based-pd-decider\\n type: disagg-profile-handler\\n- type: prefix-cache-scorer\\n- type: queue-scorer\\n- type: max-score-picker\\n- parameters:\\n scheme: https\\n type: metrics-data-source\\nschedulingProfiles:\\n- name: prefill\\n plugins:\\n - pluginRef: prefill-filter\\n - pluginRef: queue-scorer\\n weight: 2\\n - pluginRef: prefix-cache-scorer\\n weight: 3\\n - pluginRef: max-score-picker\\n- name: decode\\n plugins:\\n - pluginRef: decode-filter\\n - pluginRef: queue-scorer\\n weight: 2\\n - pluginRef: prefix-cache-scorer\\n weight: 3\\n - pluginRef: max-score-picker\\n","disable-endpoint-subset-filter":false,"drain-timeout":30000000000,"emit-endpoint-scores":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"fairness-id-metric-label-limit":1000,"feature-gates":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-cert-dir":"","metrics-client-ca-file":"","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"plugin-state-staleness-threshold":300000000000,"pool-group":"inference.networking.k8s.io","pool-name":"scheduler-v06-threshold-migration-test-inference-pool","pool-namespace":"e2e-test-llm-inference-service-40b85f95","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tls-cipher-suites":{},"tls-min-version":"","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.698573686Z","logger":"setup.trace","caller":"tracing/telemetry.go:298","body":"init OTel trace exporter","service.name":"llm-d-epp","type":"otlp","protocol":"grpc","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.699255717Z","caller":"loader/configloader.go:99","body":"Loaded raw configuration","service.name":"llm-d-epp","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: prefix-based-pd-decider, Parameters: {\\"nonCachedTokens\\":25}} {Type: disagg-profile-handler, Parameters: {\\"deciders\\":{\\"prefill\\":\\"prefix-based-pd-decider\\"}}} {Type: prefix-cache-scorer} {Type: queue-scorer} {Type: max-score-picker} {Type: metrics-data-source, Parameters: {\\"scheme\\":\\"https\\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.699309851Z","logger":"setup","caller":"runner/runner.go:757","body":"Data layer: ENABLED","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T22:35:00.699599704Z","logger":"setup","caller":"runner/runner.go:362","body":"Raw config af [e2e-llm-inference-service] ter phase one","service.name":"llm-d-epp","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"prefix-based-pd-decider","parameters":{"nonCachedTokens":25},"type":"prefix-based-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"prefix-based-pd-decider"}},"type":"disagg-profile-handler"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]},"severity_number":9}\n{"severity_text":"ERROR","timestamp":"2026-09-18T22:35:00.718035952Z","logger":"setup","caller":"runner/runner.go:387","body":"Failed to parse configuration","service.name":"llm-d-epp","error":"failed to load the configuration - configuration validation failed: plugin type \'disagg-headers-handler\' is not registered","severity_number":17,"stacktrace":"github.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).setup\\n\\t/workspace/cmd/epp/runner/runner.go:387\\ngithub.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).Run\\n\\t/workspace/cmd/epp/runner/runner.go:282\\nmain.run\\n\\t/workspace/cmd/epp/main.go:46\\nmain.main\\n\\t/workspace/cmd/epp/main.go:36\\nruntime.main\\n\\t/usr/lib/golang/src/runtime/proc.go:290"}\n' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: b161d979-4efb-4a6a-ad69-4ddad1a3c911 [e2e-llm-inference-service] resourceVersion: '36459' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - notReadyAddresses: [e2e-llm-inference-service] - ip: 10.132.0.35 [e2e-llm-inference-service] nodeName: ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr [e2e-llm-inference-service] uid: 9b30dbc3-fb8a-439e-98c4-73c6cd5f7864 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 62cb4508-d1c0-4312-ab35-cb1b98f63b44 [e2e-llm-inference-service] resourceVersion: '36823' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.34 [e2e-llm-inference-service] nodeName: ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2 [e2e-llm-inference-service] uid: b89dd3a0-382a-4e1a-b6fb-d012198819ef [e2e-llm-inference-service] - ip: 10.133.0.49 [e2e-llm-inference-service] nodeName: ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7 [e2e-llm-inference-service] uid: 0341eae0-7d5e-4ccd-a01c-846a7dccfb2e [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7 [e2e-llm-inference-service] generateName: scheduler-v06-threshold-migration-test-kserve-59b84dfb55- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 0341eae0-7d5e-4ccd-a01c-846a7dccfb2e [e2e-llm-inference-service] resourceVersion: '36821' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 59b84dfb55 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.49/23"],"mac_address":"0a:58:0a:85:00:31","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.49/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.49\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:31\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-59b84dfb55 [e2e-llm-inference-service] uid: 6afc3b3c-91f7-415e-a20d-6481da2f437f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-142-209 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6afc3b3c-91f7-415e-a20d-6481da2f437f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:54Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.49"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-v7hld [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-v7hld [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000780000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 200ms [e2e-llm-inference-service] - --inter-token-latency [e2e-llm-inference-service] - 100ms [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-v7hld [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000780000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] nodeName: ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c28,c12 [e2e-llm-inference-service] fsGroup: 1000780000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-nwkkw [e2e-llm-inference-service] - name: scheduler-v06-threshold-migration-test-kserve-dockercfg-7kq6f [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:54Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:54Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] hostIP: 10.0.142.209 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.142.209 [e2e-llm-inference-service] podIP: 10.133.0.49 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.49 [e2e-llm-inference-service] startTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-09-18T22:23:54Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar@sha256:8f710adc7cde2ba7289e68fdabd2c271737247727b0c8a9bdf00ad44bd09b34f [e2e-llm-inference-service] containerID: cri-o://5510762863da75f829979aeafd6165a2e0586fd1d8462c716016927089bd95b5 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-v7hld [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000780000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000780000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-09-18T22:23:55Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim-dev@sha256:151063768d4a46d17a30ff512d74946b20cf653a2302601a6a823727fbbe4026 [e2e-llm-inference-service] containerID: cri-o://10a20967a7a53613d1634ca2a4722220f651e896f39ce7b7618ade8924100324 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-v7hld [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000780000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000780000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2 [e2e-llm-inference-service] generateName: scheduler-v06-threshold-migration-test-kserve-prefill-64446cf4ff- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: b89dd3a0-382a-4e1a-b6fb-d012198819ef [e2e-llm-inference-service] resourceVersion: '36697' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 64446cf4ff [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.34/23"],"mac_address":"0a:58:0a:84:00:22","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.34/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.34\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:22\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-prefill-64446cf4ff [e2e-llm-inference-service] uid: 674ec277-0492-4260-bfb1-15c6853bb658 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-130-79 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"674ec277-0492-4260-bfb1-15c6853bb658"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:24:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.34"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-s9bm8 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 5s [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-s9bm8 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000780000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c28,c12 [e2e-llm-inference-service] fsGroup: 1000780000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-nwkkw [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:24:01Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:24:01Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] hostIP: 10.0.130.79 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.130.79 [e2e-llm-inference-service] podIP: 10.132.0.34 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.34 [e2e-llm-inference-service] startTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim-dev@sha256:151063768d4a46d17a30ff512d74946b20cf653a2302601a6a823727fbbe4026 [e2e-llm-inference-service] containerID: cri-o://063c955209a3a358dfd8f89e9da83764b997254f8a64049c97079cafd7fcfdf4 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-s9bm8 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000780000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000780000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr [e2e-llm-inference-service] generateName: scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 9b30dbc3-fb8a-439e-98c4-73c6cd5f7864 [e2e-llm-inference-service] resourceVersion: '45406' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 598d8f8cc9 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.35/23"],"mac_address":"0a:58:0a:84:00:23","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.35/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.35\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:23\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9 [e2e-llm-inference-service] uid: d328cd87-6144-4132-b8d9-782de2428f3a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-130-79 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d328cd87-6144-4132-b8d9-782de2428f3a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:35:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.35"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-ctjng [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- parameters:\n\ [e2e-llm-inference-service] \ nonCachedTokens: 25\n type: prefix-based-pd-decider\n- parameters:\n \ [e2e-llm-inference-service] \ deciders:\n prefill: prefix-based-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: queue-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-ctjng [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000780000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c28,c12 [e2e-llm-inference-service] fsGroup: 1000780000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-nwkkw [e2e-llm-inference-service] - name: scheduler-v06-threshold-migration-test-epp-sa-dockercfg-2wdp9 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] hostIP: 10.0.130.79 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.130.79 [e2e-llm-inference-service] podIP: 10.132.0.35 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.35 [e2e-llm-inference-service] startTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: CrashLoopBackOff [e2e-llm-inference-service] message: back-off 5m0s restarting failed container=main pod=scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr_e2e-test-llm-inference-service-40b85f95(9b30dbc3-fb8a-439e-98c4-73c6cd5f7864) [e2e-llm-inference-service] lastState: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 1 [e2e-llm-inference-service] reason: Error [e2e-llm-inference-service] message: '.name":"llm-d-epp","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"prefix-based-pd-decider","parameters":{"nonCachedTokens":25},"type":"prefix-based-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"prefix-based-pd-decider"}},"type":"disagg-profile-handler"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]},"severity_number":9} [e2e-llm-inference-service] [e2e-llm-inference-service] {"severity_text":"ERROR","timestamp":"2026-09-18T22:35:00.718035952Z","logger":"setup","caller":"runner/runner.go:387","body":"Failed [e2e-llm-inference-service] to parse configuration","service.name":"llm-d-epp","error":"failed to load [e2e-llm-inference-service] the configuration - configuration validation failed: plugin type ''disagg-headers-handler'' [e2e-llm-inference-service] is not registered","severity_number":17,"stacktrace":"github.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).setup\n\t/workspace/cmd/epp/runner/runner.go:387\ngithub.com/llm-d/llm-d-router/cmd/epp/runner.(*Runner).Run\n\t/workspace/cmd/epp/runner/runner.go:282\nmain.run\n\t/workspace/cmd/epp/main.go:46\nmain.main\n\t/workspace/cmd/epp/main.go:36\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:290"} [e2e-llm-inference-service] [e2e-llm-inference-service] ' [e2e-llm-inference-service] startedAt: '2026-09-18T22:35:00Z' [e2e-llm-inference-service] finishedAt: '2026-09-18T22:35:00Z' [e2e-llm-inference-service] containerID: cri-o://ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9 [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 7 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:c12f30f05219c4138421584f2781f32dda09373ef168b71f854861f529c0792e [e2e-llm-inference-service] containerID: cri-o://ed7eb297ca04b40a0402c9eb9a9f8b8451c93bd448ef5604d4eafbd8de35a2c9 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-ctjng [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000780000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000780000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 421cb503-7c1c-428c-a815-8ec8403f5757 [e2e-llm-inference-service] resourceVersion: '36364' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: scheduler-v06-threshold-migration-test-epp-sa-dockercfg-2wdp9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-2wdp9"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-nwkkw"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-nwkkw [e2e-llm-inference-service] - name: scheduler-v06-threshold-migration-test-epp-sa-dockercfg-2wdp9 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-nwkkw [e2e-llm-inference-service] - name: scheduler-v06-threshold-migration-test-epp-sa-dockercfg-2wdp9 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: c4d95ed4-d7d2-4151-9eac-cba551a2b8e3 [e2e-llm-inference-service] resourceVersion: '36335' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: scheduler-v06-threshold-migration-test-kserve-dockercfg-7kq6f [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"scheduler-v06-threshold-migration-test-kserve-dockercfg-7kq6f"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-nwkkw"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-nwkkw [e2e-llm-inference-service] - name: scheduler-v06-threshold-migration-test-kserve-dockercfg-7kq6f [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-nwkkw [e2e-llm-inference-service] - name: scheduler-v06-threshold-migration-test-kserve-dockercfg-7kq6f [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 2043855e-ee1d-4e1f-a0cb-a7630a495f90 [e2e-llm-inference-service] resourceVersion: '36397' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.64.66 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.64.66 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: d5d08273-2adb-430c-9d85-a5570e1c4d3f [e2e-llm-inference-service] resourceVersion: '36353' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.210.79 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.210.79 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 9f9c03d9-d12c-432a-8cbd-150f88e95d2d [e2e-llm-inference-service] resourceVersion: '36826' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 200ms [e2e-llm-inference-service] - --inter-token-latency [e2e-llm-inference-service] - 100ms [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-threshold-migration-test-kserve-59b84dfb55" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: f148f35a-b7af-420d-9e1a-4161b0bb2b3a [e2e-llm-inference-service] resourceVersion: '36701' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:24:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 5s [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:24:01Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:24:01Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:24:01Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-threshold-migration-test-kserve-prefill-64446cf4ff" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 3ef452d9-6b45-40db-bd16-cf10842c52d5 [e2e-llm-inference-service] resourceVersion: '44223' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:33:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - parameters:\n nonCachedTokens: 25\n type: prefix-based-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: prefix-based-pd-decider\n \ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- type: prefix-cache-scorer\n- type: queue-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\n- parameters:\n scheme: https\n type: metrics-data-source\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T22:33:52Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T22:33:52Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-59b84dfb55 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 6afc3b3c-91f7-415e-a20d-6481da2f437f [e2e-llm-inference-service] resourceVersion: '36825' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 59b84dfb55 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] uid: 9f9c03d9-d12c-432a-8cbd-150f88e95d2d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9f9c03d9-d12c-432a-8cbd-150f88e95d2d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 59b84dfb55 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 59b84dfb55 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 200ms [e2e-llm-inference-service] - --inter-token-latency [e2e-llm-inference-service] - 100ms [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-prefill-64446cf4ff [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 674ec277-0492-4260-bfb1-15c6853bb658 [e2e-llm-inference-service] resourceVersion: '36700' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 64446cf4ff [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-prefill [e2e-llm-inference-service] uid: f148f35a-b7af-420d-9e1a-4161b0bb2b3a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f148f35a-b7af-420d-9e1a-4161b0bb2b3a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:24:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 64446cf4ff [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 64446cf4ff [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim-dev:1d5ad96 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --force-dummy-tokenizer [e2e-llm-inference-service] - --time-to-first-token [e2e-llm-inference-service] - 5s [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: d328cd87-6144-4132-b8d9-782de2428f3a [e2e-llm-inference-service] resourceVersion: '36415' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 598d8f8cc9 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-router-scheduler [e2e-llm-inference-service] uid: 3ef452d9-6b45-40db-bd16-cf10842c52d5 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3ef452d9-6b45-40db-bd16-cf10842c52d5"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 598d8f8cc9 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 598d8f8cc9 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - parameters:\n nonCachedTokens: 25\n type: prefix-based-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: prefix-based-pd-decider\n \ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- type: prefix-cache-scorer\n- type: queue-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\n- parameters:\n scheme: https\n type: metrics-data-source\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-prometheus-scraping [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 73a14e44-4ad2-4735-8cc5-0fe68d3ad9c8 [e2e-llm-inference-service] resourceVersion: '36537' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:53Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llm-monitoring [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: networking.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:53Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:ingress: {} [e2e-llm-inference-service] f:podSelector: {} [e2e-llm-inference-service] f:policyTypes: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] podSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ingress: [e2e-llm-inference-service] - ports: [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] from: [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: openshift-monitoring [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: openshift-user-workload-monitoring [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: redhat-ods-monitoring [e2e-llm-inference-service] - ports: [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] from: [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: openshift-ingress [e2e-llm-inference-service] - from: [e2e-llm-inference-service] - podSelector: {} [e2e-llm-inference-service] policyTypes: [e2e-llm-inference-service] - Ingress [e2e-llm-inference-service] apiVersion: networking.k8s.io/v1 [e2e-llm-inference-service] kind: NetworkPolicy [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 51e8d87a-a83c-42e1-bcad-446fa32cf041 [e2e-llm-inference-service] resourceVersion: '36392' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 05a7cd01-5c28-440f-bd40-ccf9ee9ebbde [e2e-llm-inference-service] resourceVersion: '36343' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 37f1453f-3eb0-4cb1-b6da-55468d22d2b4 [e2e-llm-inference-service] resourceVersion: '36390' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - patch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - events [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: af434989-caa0-42ac-8894-54d3c54ecf5d [e2e-llm-inference-service] resourceVersion: '36339' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-service-jf9sd [e2e-llm-inference-service] generateName: scheduler-v06-threshold-migration-test-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 51f25451-a7eb-4018-bd2e-a55cc76f4606 [e2e-llm-inference-service] resourceVersion: '36458' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: scheduler-v06-threshold-migration-test-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-service [e2e-llm-inference-service] uid: 2043855e-ee1d-4e1f-a0cb-a7630a495f90 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2043855e-ee1d-4e1f-a0cb-a7630a495f90"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.35 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] serving: false [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-router-schedt2dgr [e2e-llm-inference-service] uid: 9b30dbc3-fb8a-439e-98c4-73c6cd5f7864 [e2e-llm-inference-service] nodeName: ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc47fb2 [e2e-llm-inference-service] generateName: scheduler-v06-threshold-migration-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 5f1ca549-14fb-4bad-86ce-e089f581ea30 [e2e-llm-inference-service] resourceVersion: '36824' [e2e-llm-inference-service] generation: 5 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] uid: d5d08273-2adb-430c-9d85-a5570e1c4d3f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:24:13Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d5d08273-2adb-430c-9d85-a5570e1c4d3f"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.34 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2 [e2e-llm-inference-service] uid: b89dd3a0-382a-4e1a-b6fb-d012198819ef [e2e-llm-inference-service] nodeName: ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.49 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7 [e2e-llm-inference-service] uid: 0341eae0-7d5e-4ccd-a01c-846a7dccfb2e [e2e-llm-inference-service] nodeName: ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 51e8d87a-a83c-42e1-bcad-446fa32cf041 [e2e-llm-inference-service] resourceVersion: '36392' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-40b85f95:scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 05a7cd01-5c28-440f-bd40-ccf9ee9ebbde [e2e-llm-inference-service] resourceVersion: '36343' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-40b85f95:scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: 37f1453f-3eb0-4cb1-b6da-55468d22d2b4 [e2e-llm-inference-service] resourceVersion: '36390' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - patch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - events [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] uid: af434989-caa0-42ac-8894-54d3c54ecf5d [e2e-llm-inference-service] resourceVersion: '36339' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T22:23:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:24:18Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:24:18Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:24:19Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36910' [e2e-llm-inference-service] uid: 7301bd99-9ca4-4efd-aa35-417b230d290c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:24:18Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:23:53Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:24:18Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:24:18Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:24:19Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36910' [e2e-llm-inference-service] uid: 7301bd99-9ca4-4efd-aa35-417b230d290c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-40b85f95/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:24:18Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:23:53Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:24:18Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36892' [e2e-llm-inference-service] uid: d4772a5d-4a95-49e8-8ff3-8c23d5a41a35 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:24:18Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:24:18Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:53Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:23:53Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T22:23:56Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36645' [e2e-llm-inference-service] uid: abf8ceae-795e-45aa-80a1-fd9e986fd3e5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:23:55Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T22:23:56Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36438' [e2e-llm-inference-service] uid: 45d2baee-7204-4c84-b92e-b93efee8f8df [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-threshold-migration-test-epp-service.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-threshold-migration-test-epp-service.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:24:22Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:24:22Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36943' [e2e-llm-inference-service] uid: baf01415-f7be-4d0d-89e3-db034e62fea2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-threshold-migration-test-inference-po-ip-32d3e584.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-threshold-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36482' [e2e-llm-inference-service] uid: 9c05205f-5e70-442a-8639-2579ea0bf845 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-threshold-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-threshold-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36438' [e2e-llm-inference-service] uid: 45d2baee-7204-4c84-b92e-b93efee8f8df [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-threshold-migration-test-epp-service.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-threshold-migration-test-epp-service.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:24:22Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:24:22Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36943' [e2e-llm-inference-service] uid: baf01415-f7be-4d0d-89e3-db034e62fea2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-threshold-migration-test-inference-po-ip-32d3e584.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-threshold-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36482' [e2e-llm-inference-service] uid: 9c05205f-5e70-442a-8639-2579ea0bf845 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-threshold-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-threshold-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36438' [e2e-llm-inference-service] uid: 45d2baee-7204-4c84-b92e-b93efee8f8df [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-threshold-migration-test-epp-service.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-threshold-migration-test-epp-service.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:24:22Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:24:22Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36943' [e2e-llm-inference-service] uid: baf01415-f7be-4d0d-89e3-db034e62fea2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-threshold-migration-test-inference-po-ip-32d3e584.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-threshold-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:23:52Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36482' [e2e-llm-inference-service] uid: 9c05205f-5e70-442a-8639-2579ea0bf845 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-threshold-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-threshold-migration-test-kserve-workload-svc.e2e-test-llm-inference-service-40b85f95.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b03f6e1b-6144-488d-a95f-be80dc490c60"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T22:23:51Z' [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] uid: b03f6e1b-6144-488d-a95f-be80dc490c60 [e2e-llm-inference-service] resourceVersion: '36409' [e2e-llm-inference-service] uid: 0f144418-214d-4e8a-b381-9cfb52045503 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:38:32Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 59b84dfb55 [e2e-llm-inference-service] timestamp: '2026-09-18T22:38:13Z' [e2e-llm-inference-service] window: 13.379s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 11920920n [e2e-llm-inference-service] memory: 22968Ki [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 2698610n [e2e-llm-inference-service] memory: 20332Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: scheduler-v06-threshold-migration-test-kserve-prefill-64444m4d2 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-40b85f95 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T22:38:32Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-threshold-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 64446cf4ff [e2e-llm-inference-service] timestamp: '2026-09-18T22:38:18Z' [e2e-llm-inference-service] window: 17.138s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 10847240n [e2e-llm-inference-service] memory: 21324Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-09-18T22:38:32.677797] end - ❌ 901.482s: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:24:23Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:190 Skipping deletion of namespace e2e-test-llm-inference-service-40b85f95 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] ___ test_llm_tls_resources[router-managed-workload-pd-cpu-model-fb-opt-125m] ___ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name='tls... {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] service_name="tls-verification-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_tls_resources(test_case: TestCase): [e2e-llm-inference-service] """Verify that TLS-related resources (DestinationRules, cert secrets, service port) [e2e-llm-inference-service] are correctly present or absent based on the enableLLMInferenceServiceTLS flag.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] tls_enabled, _, _ = _get_tls_config() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] wait_for_model_response(kserve_client, test_case, test_case.wait_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] tls_enabled, tls_min_version, tls_cipher_suites = _get_tls_config() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] "LLMInferenceService TLS config: enabled=%s, min_version=%s, cipher_suites=%s", [e2e-llm-inference-service] tls_enabled, [e2e-llm-inference-service] tls_min_version, [e2e-llm-inference-service] tls_cipher_suites, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > wait_for( [e2e-llm-inference-service] lambda: _verify_tls_arguments( [e2e-llm-inference-service] service_name, [e2e-llm-inference-service] test_case.namespace, [e2e-llm-inference-service] tls_min_version, [e2e-llm-inference-service] tls_cipher_suites, [e2e-llm-inference-service] ), [e2e-llm-inference-service] timeout=test_case.wait_timeout, [e2e-llm-inference-service] interval=2.0, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:206: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = . at 0x7f4c960d6b60> [e2e-llm-inference-service] timeout = 900, interval = 2.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] ^^^^^^^^^^^^^^ [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1443: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] > lambda: _verify_tls_arguments( [e2e-llm-inference-service] service_name, [e2e-llm-inference-service] test_case.namespace, [e2e-llm-inference-service] tls_min_version, [e2e-llm-inference-service] tls_cipher_suites, [e2e-llm-inference-service] ), [e2e-llm-inference-service] timeout=test_case.wait_timeout, [e2e-llm-inference-service] interval=2.0, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:207: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = 'tls-verification-test' [e2e-llm-inference-service] namespace = 'e2e-test-llm-tls-resources-e4cad0c3', tls_min_version = '' [e2e-llm-inference-service] tls_cipher_suites = '' [e2e-llm-inference-service] [e2e-llm-inference-service] def _verify_tls_arguments(service_name, namespace, tls_min_version, tls_cipher_suites): [e2e-llm-inference-service] """Assert that the TLS profile is propagated to the EPP and routing sidecar.""" [e2e-llm-inference-service] commands = _get_container_commands(namespace, service_name) [e2e-llm-inference-service] assert commands["epp"], "Expected to find the EPP container command" [e2e-llm-inference-service] assert commands["sidecar"], "Expected to find the routing sidecar command" [e2e-llm-inference-service] assert commands["vllm"], "Expected to find the vLLM workload command" [e2e-llm-inference-service] [e2e-llm-inference-service] expected_args = { [e2e-llm-inference-service] "--tls-min-version": tls_min_version, [e2e-llm-inference-service] "--tls-cipher-suites": tls_cipher_suites, [e2e-llm-inference-service] } [e2e-llm-inference-service] for component in ("epp", "sidecar"): [e2e-llm-inference-service] for flag, value in expected_args.items(): [e2e-llm-inference-service] matching_args = [ [e2e-llm-inference-service] arg [e2e-llm-inference-service] for command in commands[component] [e2e-llm-inference-service] for arg in command [e2e-llm-inference-service] if arg.startswith(f"{flag}=") [e2e-llm-inference-service] ] [e2e-llm-inference-service] if value: [e2e-llm-inference-service] assert f"{flag}={value}" in matching_args, ( [e2e-llm-inference-service] f"Expected {component} to receive {flag}={value}, " [e2e-llm-inference-service] f"got: {matching_args}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] assert not matching_args, ( [e2e-llm-inference-service] f"Expected {component} to omit {flag}, got: {matching_args}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] vllm_command = " ".join(arg for command in commands["vllm"] for arg in command) [e2e-llm-inference-service] if tls_cipher_suites: [e2e-llm-inference-service] for cipher_suite in OPENSSL_TLS_CIPHER_SUITES: [e2e-llm-inference-service] assert cipher_suite in vllm_command, ( [e2e-llm-inference-service] f"Expected vLLM command to contain {cipher_suite}, got: {vllm_command}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] assert "--ssl-ciphers" in vllm_command [e2e-llm-inference-service] else: [e2e-llm-inference-service] > assert "--ssl-ciphers" not in vllm_command [e2e-llm-inference-service] E assert '--ssl-ciphers' not in '/bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\n [e2e-llm-inference-service] for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there\'s a tie, prefer index 3 as it\'s most common in SR-IOV setups\n if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_index_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n# ODH supports older accelerator images, so only pass it when available.\n\n\n# --shutdo [e2e-llm-inference-service] wn-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Decode is the KV consumer; without a connector here it recomputes prefill\'s KV.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_consumer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \\\n --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \\\n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSER [e2e-llm-inference-service] VE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there\'s a tie, prefer index 3 as it\'s most common in SR-IOV setups\n if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_in [e2e-llm-inference-service] dex_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n# ODH supports older accelerator images, so only pass it when available.\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Prefill is the KV producer; without a connector here decode has nothing to fetch.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_producer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \\\n --served-model-name "facebook/opt-125m" \\\n --port 8000 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verif [e2e-llm-inference-service] ication-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" --' [e2e-llm-inference-service] E [e2e-llm-inference-service] E '--ssl-ciphers' is contained here: [e2e-llm-inference-service] E /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] E if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] E ARCH="$(arch)" [e2e-llm-inference-service] E case "${ARCH}" in [e2e-llm-inference-service] E ppc64le) [e2e-llm-inference-service] E if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] E . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] E export PATH [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E ;; [e2e-llm-inference-service] E s390x) [e2e-llm-inference-service] E export FLEX_DEVICE=VF [e2e-llm-inference-service] E if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] E source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] E if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] E if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] E export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] E rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] E . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] E export PATH [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E ;; [e2e-llm-inference-service] E x86_64) [e2e-llm-inference-service] E export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] E ;; [e2e-llm-inference-service] E esac [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] E source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] E echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] E grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] E grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] E [e2e-llm-inference-service] E cat /proc/driver/nvidia/params [e2e-llm-inference-service] E [e2e-llm-inference-service] E KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] E [e2e-llm-inference-service] E echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] E active_hcas=() [e2e-llm-inference-service] E # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] E for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] E # Ensure it's a directory before proceeding [e2e-llm-inference-service] E if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] E hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] E port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] E type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] E [e2e-llm-inference-service] E echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] E if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] E echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] E active_hcas+=("$hca_name") [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E done [e2e-llm-inference-service] E [e2e-llm-inference-service] E # Check if we found any active HCAs [e2e-llm-inference-service] E if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] E # Join the array elements with a comma [e2e-llm-inference-service] E hca_port_pairs=() [e2e-llm-inference-service] E for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] E hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] E done [e2e-llm-inference-service] E [e2e-llm-inference-service] E active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] E hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] E echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] E export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] E export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] E export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] E [e2e-llm-inference-service] E echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] E echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] E echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] E echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] E [e2e-llm-inference-service] E # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] E declare -A gid_index_count [e2e-llm-inference-service] E declare -A hca_gid_index [e2e-llm-inference-service] E [e2e-llm-inference-service] E for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] E echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] E [e2e-llm-inference-service] E # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] E for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] E if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] E idx=$(basename "$tpath") [e2e-llm-inference-service] E gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] E # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] E if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] E gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] E echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] E hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] E gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] E break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E done [e2e-llm-inference-service] E done [e2e-llm-inference-service] E [e2e-llm-inference-service] E # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] E best_gid_index="" [e2e-llm-inference-service] E max_count=0 [e2e-llm-inference-service] E for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] E count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] E echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] E if [ $count -gt $max_count ]; then [e2e-llm-inference-service] E max_count=$count [e2e-llm-inference-service] E best_gid_index="$idx" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E done [e2e-llm-inference-service] E [e2e-llm-inference-service] E # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] E if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] E echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] E # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] E if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] E best_gid_index="3" [e2e-llm-inference-service] E echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] E if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] E echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] E export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] E export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] E echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] E elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] E echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] E [e2e-llm-inference-service] E export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] E export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] E export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] E [e2e-llm-inference-service] E echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] E # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] E ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] E VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] E echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] E if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] E ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] E [e2e-llm-inference-service] E # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] E ? +++++++++++++ [e2e-llm-inference-service] E # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] E [e2e-llm-inference-service] E [e2e-llm-inference-service] E # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] E SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] E if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] E SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] E KV_TRANSFER_ARGS="" [e2e-llm-inference-service] E if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] E # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] E if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] E KV_TRANSFER_ARGS="" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] E # Decode is the KV consumer; without a connector here it recomputes prefill's KV. [e2e-llm-inference-service] E if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] E # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] E # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] E NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] E if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] E KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\"}'" [e2e-llm-inference-service] E echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)" [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] E --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \ [e2e-llm-inference-service] E --port 8001 \ [e2e-llm-inference-service] E --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] E ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] E ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] E ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] E --enable-ssl-refresh \ [e2e-llm-inference-service] E --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] E --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] E ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] E $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] E if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] E ARCH="$(arch)" [e2e-llm-inference-service] E case "${ARCH}" in [e2e-llm-inference-service] E ppc64le) [e2e-llm-inference-service] E if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] E . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] E export PATH [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E ;; [e2e-llm-inference-service] E s390x) [e2e-llm-inference-service] E export FLEX_DEVICE=VF [e2e-llm-inference-service] E if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] E source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] E if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] E if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] E export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] E rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] E . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] E export PATH [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E ;; [e2e-llm-inference-service] E x86_64) [e2e-llm-inference-service] E export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] E ;; [e2e-llm-inference-service] E esac [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] E source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] E echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] E grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] E grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] E [e2e-llm-inference-service] E cat /proc/driver/nvidia/params [e2e-llm-inference-service] E [e2e-llm-inference-service] E KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] E [e2e-llm-inference-service] E echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] E active_hcas=() [e2e-llm-inference-service] E # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] E for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] E # Ensure it's a directory before proceeding [e2e-llm-inference-service] E if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] E hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] E port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] E type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] E [e2e-llm-inference-service] E echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] E if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] E echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] E active_hcas+=("$hca_name") [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E done [e2e-llm-inference-service] E [e2e-llm-inference-service] E # Check if we found any active HCAs [e2e-llm-inference-service] E if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] E # Join the array elements with a comma [e2e-llm-inference-service] E hca_port_pairs=() [e2e-llm-inference-service] E for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] E hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] E done [e2e-llm-inference-service] E [e2e-llm-inference-service] E active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] E hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] E echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] E export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] E export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] E export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] E [e2e-llm-inference-service] E echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] E echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] E echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] E echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] E [e2e-llm-inference-service] E # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] E declare -A gid_index_count [e2e-llm-inference-service] E declare -A hca_gid_index [e2e-llm-inference-service] E [e2e-llm-inference-service] E for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] E echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] E [e2e-llm-inference-service] E # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] E for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] E if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] E idx=$(basename "$tpath") [e2e-llm-inference-service] E gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] E # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] E if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] E gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] E echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] E hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] E gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] E break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E done [e2e-llm-inference-service] E done [e2e-llm-inference-service] E [e2e-llm-inference-service] E # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] E best_gid_index="" [e2e-llm-inference-service] E max_count=0 [e2e-llm-inference-service] E for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] E count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] E echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] E if [ $count -gt $max_count ]; then [e2e-llm-inference-service] E max_count=$count [e2e-llm-inference-service] E best_gid_index="$idx" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E done [e2e-llm-inference-service] E [e2e-llm-inference-service] E # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] E if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] E echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] E # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] E if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] E best_gid_index="3" [e2e-llm-inference-service] E echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] E if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] E echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] E export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] E export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] E echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] E elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] E echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] E [e2e-llm-inference-service] E export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] E export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] E export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] E [e2e-llm-inference-service] E echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] E # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] E ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] E VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] E echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] E if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] E ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] E [e2e-llm-inference-service] E # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] E # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] E [e2e-llm-inference-service] E [e2e-llm-inference-service] E # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] E SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] E if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] E SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] E KV_TRANSFER_ARGS="" [e2e-llm-inference-service] E if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] E # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] E if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] E KV_TRANSFER_ARGS="" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] E # Prefill is the KV producer; without a connector here decode has nothing to fetch. [e2e-llm-inference-service] E if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] E # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] E # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] E NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] E if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] E KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\"}'" [e2e-llm-inference-service] E echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)" [e2e-llm-inference-service] E else [e2e-llm-inference-service] E echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E fi [e2e-llm-inference-service] E [e2e-llm-inference-service] E eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] E --served-model-name "facebook/opt-125m" \ [e2e-llm-inference-service] E --port 8000 \ [e2e-llm-inference-service] E --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] E ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] E ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] E ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] E --enable-ssl-refresh \ [e2e-llm-inference-service] E --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] E --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] E ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] E $@" -- [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:340: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-tls-resources-e4cad0c3 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig router-managed-tls-verification-86160844 in namespace e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig router-managed-tls-verification-86160844 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig router-managed-tls-verification-86160844 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig workload-pd-cpu-tls-verificatio-cc747c7a in namespace e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-tls-verificatio-cc747c7a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-tls-verificatio-cc747c7a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2119 Checking LLMInferenceServiceConfig model-fb-opt-125m-tls-verificat-7c5fca28 in namespace e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2145 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-tls-verificat-7c5fca28 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2155 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-tls-verificat-7c5fca28 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_tls_resources] [2026-09-18T23:09:18.706140] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name='tls-verification-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f4c9699dee0>, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-tls-resources-e4cad0c3', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-09-18T23:09:18.753426] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-09-18T23:09:18.857804] end - ✅ in 0.104s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-09-18T23:09:18.858014] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T23:09:48Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route]', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'message': 'Inference Pool e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route]', 'reason': 'WaitingForGateway', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'message': 'The following HTTPRoutes exist but no Gateway controller has accepted them yet: [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route]', 'reason': 'WaitingForGateway', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route: Accepted=True (reason "Accepted", message "Route was valid")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T23:10:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T23:10:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T23:10:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T23:10:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T23:10:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T23:10:21Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T23:10:21Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 13: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-09-18T22:05:20Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T23:10:12Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:11:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:48Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T23:10:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T23:10:21Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T23:10:21Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T23:09:58Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-09-18T23:12:07.456794] end - ✅ in 168.598s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-09-18T23:12:07.456962] start - args=(, TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-fb-opt-125m'], prompt='KServe is a', service_name='tls-verification-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=, response_assertion=.response_assertion at 0x7f4c9699dee0>, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-tls-resources-e4cad0c3', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-09-18T23:12:07.457255] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-09-18T23:12:07.463965] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1313 Calling LLM service at http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1332 Model response is 200: {"choices":[{"finish_reason":"length","index":0,"logprobs":null,"prompt_logprobs":null,"prompt_token_ids":null,"stop_reason":null,"text":" web site for individuals interested in improving the health of their homes and communities. KServe allows individuals in","token_ids":null}],"created":1789773130,"id":"cmpl-8ec8e3a4-57ee-472a-8673-b0732998f701","kv_transfer_params":null,"model":"facebook/opt-125m","object":"text_completion","service_tier":null,"system_fingerprint":null,"usage":{"completion_to [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_model_response] [2026-09-18T23:12:14.019755] end - ✅ in 6.563s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2170 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:test_llm_tls.py:199 LLMInferenceService TLS config: enabled=True, min_version=, cipher_suites= [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: assert '--ssl-ciphers' not in '/bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs\n d [e2e-llm-inference-service] eclare -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there\'s a tie, prefer index 3 as it\'s most common in SR-IOV setups\n if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_index_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n# ODH supports older accele [e2e-llm-inference-service] rator images, so only pass it when available.\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Decode is the KV consumer; without a connector here it recomputes prefill\'s KV.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_consumer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \\\n --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \\\n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\n cat /proc/dr [e2e-llm-inference-service] iver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there\'s a tie, prefer index 3 as it\'s most common in SR-IOV setups\n [e2e-llm-inference-service] if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_index_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n# ODH supports older accelerator images, so only pass it when available.\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Prefill is the KV producer; without a connector here decode has nothing to fetch.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_producer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \\\n --served-model-name "facebook/opt-125m" \\\n --port 8000 \\\ [e2e-llm-inference-service] n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" --' [e2e-llm-inference-service] [e2e-llm-inference-service] '--ssl-ciphers' is contained here: [e2e-llm-inference-service] /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] ARCH="$(arch)" [e2e-llm-inference-service] case "${ARCH}" in [e2e-llm-inference-service] ppc64le) [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] s390x) [e2e-llm-inference-service] export FLEX_DEVICE=VF [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] fi [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] x86_64) [e2e-llm-inference-service] export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] ;; [e2e-llm-inference-service] esac [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] [e2e-llm-inference-service] cat /proc/driver/nvidia/params [e2e-llm-inference-service] [e2e-llm-inference-service] KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] active_hcas=() [e2e-llm-inference-service] # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] # Ensure it's a directory before proceeding [e2e-llm-inference-service] if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] active_hcas+=("$hca_name") [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if we found any active HCAs [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] # Join the array elements with a comma [e2e-llm-inference-service] hca_port_pairs=() [e2e-llm-inference-service] for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] [e2e-llm-inference-service] # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] declare -A gid_index_count [e2e-llm-inference-service] declare -A hca_gid_index [e2e-llm-inference-service] [e2e-llm-inference-service] for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] [e2e-llm-inference-service] # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] idx=$(basename "$tpath") [e2e-llm-inference-service] gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] best_gid_index="" [e2e-llm-inference-service] max_count=0 [e2e-llm-inference-service] for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] if [ $count -gt $max_count ]; then [e2e-llm-inference-service] max_count=$count [e2e-llm-inference-service] best_gid_index="$idx" [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] best_gid_index="3" [e2e-llm-inference-service] echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] [e2e-llm-inference-service] export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] fi [e2e-llm-inference-service] echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] [e2e-llm-inference-service] # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] ? +++++++++++++ [e2e-llm-inference-service] # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] fi [e2e-llm-inference-service] # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] # Decode is the KV consumer; without a connector here it recomputes prefill's KV. [e2e-llm-inference-service] if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\"}'" [e2e-llm-inference-service] echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \ [e2e-llm-inference-service] --port 8001 \ [e2e-llm-inference-service] --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] --enable-ssl-refresh \ [e2e-llm-inference-service] --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] ARCH="$(arch)" [e2e-llm-inference-service] case "${ARCH}" in [e2e-llm-inference-service] ppc64le) [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] s390x) [e2e-llm-inference-service] export FLEX_DEVICE=VF [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] fi [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] x86_64) [e2e-llm-inference-service] export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] ;; [e2e-llm-inference-service] esac [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] [e2e-llm-inference-service] cat /proc/driver/nvidia/params [e2e-llm-inference-service] [e2e-llm-inference-service] KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] active_hcas=() [e2e-llm-inference-service] # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] # Ensure it's a directory before proceeding [e2e-llm-inference-service] if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] active_hcas+=("$hca_name") [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if we found any active HCAs [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] # Join the array elements with a comma [e2e-llm-inference-service] hca_port_pairs=() [e2e-llm-inference-service] for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] [e2e-llm-inference-service] # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] declare -A gid_index_count [e2e-llm-inference-service] declare -A hca_gid_index [e2e-llm-inference-service] [e2e-llm-inference-service] for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] [e2e-llm-inference-service] # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] idx=$(basename "$tpath") [e2e-llm-inference-service] gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] best_gid_index="" [e2e-llm-inference-service] max_count=0 [e2e-llm-inference-service] for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] if [ $count -gt $max_count ]; then [e2e-llm-inference-service] max_count=$count [e2e-llm-inference-service] best_gid_index="$idx" [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] best_gid_index="3" [e2e-llm-inference-service] echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] [e2e-llm-inference-service] export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] fi [e2e-llm-inference-service] echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] [e2e-llm-inference-service] # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] fi [e2e-llm-inference-service] # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] # Prefill is the KV producer; without a connector here decode has nothing to fetch. [e2e-llm-inference-service] if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\"}'" [e2e-llm-inference-service] echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] --served-model-name "facebook/opt-125m" \ [e2e-llm-inference-service] --port 8000 \ [e2e-llm-inference-service] --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] --enable-ssl-refresh \ [e2e-llm-inference-service] --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] $@" -- [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 14: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-09-18T22:05:20Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 15: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-09-18T22:05:20Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 16: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-09-18T22:05:20Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1447 Timed out waiting: assert '--ssl-ciphers' not in '/bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCA [e2e-llm-inference-service] s\n declare -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there\'s a tie, prefer index 3 as it\'s most common in SR-IOV setups\n if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_index_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n# ODH supports ol [e2e-llm-inference-service] der accelerator images, so only pass it when available.\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Decode is the KV consumer; without a connector here it recomputes prefill\'s KV.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_consumer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \\\n --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \\\n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\n ca [e2e-llm-inference-service] t /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there\'s a tie, prefer index 3 as it\'s most common in SR-IOV [e2e-llm-inference-service] setups\n if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_index_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n# ODH supports older accelerator images, so only pass it when available.\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Prefill is the KV producer; without a connector here decode has nothing to fetch.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_producer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \\\n --served-model-name "facebook/opt-125m" \\\n --por [e2e-llm-inference-service] t 8000 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" --' [e2e-llm-inference-service] [e2e-llm-inference-service] '--ssl-ciphers' is contained here: [e2e-llm-inference-service] /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] ARCH="$(arch)" [e2e-llm-inference-service] case "${ARCH}" in [e2e-llm-inference-service] ppc64le) [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] s390x) [e2e-llm-inference-service] export FLEX_DEVICE=VF [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] fi [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] x86_64) [e2e-llm-inference-service] export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] ;; [e2e-llm-inference-service] esac [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] [e2e-llm-inference-service] cat /proc/driver/nvidia/params [e2e-llm-inference-service] [e2e-llm-inference-service] KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] active_hcas=() [e2e-llm-inference-service] # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] # Ensure it's a directory before proceeding [e2e-llm-inference-service] if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] active_hcas+=("$hca_name") [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if we found any active HCAs [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] # Join the array elements with a comma [e2e-llm-inference-service] hca_port_pairs=() [e2e-llm-inference-service] for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] [e2e-llm-inference-service] # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] declare -A gid_index_count [e2e-llm-inference-service] declare -A hca_gid_index [e2e-llm-inference-service] [e2e-llm-inference-service] for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] [e2e-llm-inference-service] # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] idx=$(basename "$tpath") [e2e-llm-inference-service] gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] best_gid_index="" [e2e-llm-inference-service] max_count=0 [e2e-llm-inference-service] for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] if [ $count -gt $max_count ]; then [e2e-llm-inference-service] max_count=$count [e2e-llm-inference-service] best_gid_index="$idx" [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] best_gid_index="3" [e2e-llm-inference-service] echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] [e2e-llm-inference-service] export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] fi [e2e-llm-inference-service] echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] [e2e-llm-inference-service] # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] ? +++++++++++++ [e2e-llm-inference-service] # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] fi [e2e-llm-inference-service] # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] # Decode is the KV consumer; without a connector here it recomputes prefill's KV. [e2e-llm-inference-service] if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\"}'" [e2e-llm-inference-service] echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \ [e2e-llm-inference-service] --port 8001 \ [e2e-llm-inference-service] --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] --enable-ssl-refresh \ [e2e-llm-inference-service] --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] ARCH="$(arch)" [e2e-llm-inference-service] case "${ARCH}" in [e2e-llm-inference-service] ppc64le) [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] s390x) [e2e-llm-inference-service] export FLEX_DEVICE=VF [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] fi [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] x86_64) [e2e-llm-inference-service] export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] ;; [e2e-llm-inference-service] esac [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] [e2e-llm-inference-service] cat /proc/driver/nvidia/params [e2e-llm-inference-service] [e2e-llm-inference-service] KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] active_hcas=() [e2e-llm-inference-service] # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] # Ensure it's a directory before proceeding [e2e-llm-inference-service] if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] active_hcas+=("$hca_name") [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if we found any active HCAs [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] # Join the array elements with a comma [e2e-llm-inference-service] hca_port_pairs=() [e2e-llm-inference-service] for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] [e2e-llm-inference-service] # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] declare -A gid_index_count [e2e-llm-inference-service] declare -A hca_gid_index [e2e-llm-inference-service] [e2e-llm-inference-service] for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] [e2e-llm-inference-service] # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] idx=$(basename "$tpath") [e2e-llm-inference-service] gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] best_gid_index="" [e2e-llm-inference-service] max_count=0 [e2e-llm-inference-service] for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] if [ $count -gt $max_count ]; then [e2e-llm-inference-service] max_count=$count [e2e-llm-inference-service] best_gid_index="$idx" [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] best_gid_index="3" [e2e-llm-inference-service] echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] [e2e-llm-inference-service] export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] fi [e2e-llm-inference-service] echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] [e2e-llm-inference-service] # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] fi [e2e-llm-inference-service] # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] # Prefill is the KV producer; without a connector here decode has nothing to fetch. [e2e-llm-inference-service] if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\"}'" [e2e-llm-inference-service] echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] --served-model-name "facebook/opt-125m" \ [e2e-llm-inference-service] --port 8000 \ [e2e-llm-inference-service] --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] --enable-ssl-refresh \ [e2e-llm-inference-service] --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] $@" -- [e2e-llm-inference-service] ERROR e2e.llmisvc.test_llm_tls:test_llm_tls.py:219 Failed TLS verification for tls-verification-test: assert '--ssl-ciphers' not in '/bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GI [e2e-llm-inference-service] D index across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there\'s a tie, prefer index 3 as it\'s most common in SR-IOV setups\n if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_index_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#3093 [e2e-llm-inference-service] 7).\n# ODH supports older accelerator images, so only pass it when available.\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Decode is the KV consumer; without a connector here it recomputes prefill\'s KV.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_consumer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \\\n --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \\\n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types [e2e-llm-inference-service] /* 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there\'s a tie, prefer index 3 as it\'s [e2e-llm-inference-service] most common in SR-IOV setups\n if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_index_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n# ODH supports older accelerator images, so only pass it when available.\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Prefill is the KV producer; without a connector here decode has nothing to fetch.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_producer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \\\n --served-model-name "facebook [e2e-llm-inference-service] /opt-125m" \\\n --port 8000 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" --' [e2e-llm-inference-service] [e2e-llm-inference-service] '--ssl-ciphers' is contained here: [e2e-llm-inference-service] /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] ARCH="$(arch)" [e2e-llm-inference-service] case "${ARCH}" in [e2e-llm-inference-service] ppc64le) [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] s390x) [e2e-llm-inference-service] export FLEX_DEVICE=VF [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] fi [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] x86_64) [e2e-llm-inference-service] export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] ;; [e2e-llm-inference-service] esac [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] [e2e-llm-inference-service] cat /proc/driver/nvidia/params [e2e-llm-inference-service] [e2e-llm-inference-service] KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] active_hcas=() [e2e-llm-inference-service] # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] # Ensure it's a directory before proceeding [e2e-llm-inference-service] if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] active_hcas+=("$hca_name") [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if we found any active HCAs [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] # Join the array elements with a comma [e2e-llm-inference-service] hca_port_pairs=() [e2e-llm-inference-service] for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] [e2e-llm-inference-service] # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] declare -A gid_index_count [e2e-llm-inference-service] declare -A hca_gid_index [e2e-llm-inference-service] [e2e-llm-inference-service] for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] [e2e-llm-inference-service] # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] idx=$(basename "$tpath") [e2e-llm-inference-service] gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] best_gid_index="" [e2e-llm-inference-service] max_count=0 [e2e-llm-inference-service] for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] if [ $count -gt $max_count ]; then [e2e-llm-inference-service] max_count=$count [e2e-llm-inference-service] best_gid_index="$idx" [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] best_gid_index="3" [e2e-llm-inference-service] echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] [e2e-llm-inference-service] export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] fi [e2e-llm-inference-service] echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] [e2e-llm-inference-service] # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] ? +++++++++++++ [e2e-llm-inference-service] # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] fi [e2e-llm-inference-service] # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] # Decode is the KV consumer; without a connector here it recomputes prefill's KV. [e2e-llm-inference-service] if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\"}'" [e2e-llm-inference-service] echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \ [e2e-llm-inference-service] --port 8001 \ [e2e-llm-inference-service] --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] --enable-ssl-refresh \ [e2e-llm-inference-service] --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] ARCH="$(arch)" [e2e-llm-inference-service] case "${ARCH}" in [e2e-llm-inference-service] ppc64le) [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] s390x) [e2e-llm-inference-service] export FLEX_DEVICE=VF [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] fi [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] x86_64) [e2e-llm-inference-service] export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] ;; [e2e-llm-inference-service] esac [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] [e2e-llm-inference-service] cat /proc/driver/nvidia/params [e2e-llm-inference-service] [e2e-llm-inference-service] KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] active_hcas=() [e2e-llm-inference-service] # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] # Ensure it's a directory before proceeding [e2e-llm-inference-service] if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] active_hcas+=("$hca_name") [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if we found any active HCAs [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] # Join the array elements with a comma [e2e-llm-inference-service] hca_port_pairs=() [e2e-llm-inference-service] for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] [e2e-llm-inference-service] # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] declare -A gid_index_count [e2e-llm-inference-service] declare -A hca_gid_index [e2e-llm-inference-service] [e2e-llm-inference-service] for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] [e2e-llm-inference-service] # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] idx=$(basename "$tpath") [e2e-llm-inference-service] gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] best_gid_index="" [e2e-llm-inference-service] max_count=0 [e2e-llm-inference-service] for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] if [ $count -gt $max_count ]; then [e2e-llm-inference-service] max_count=$count [e2e-llm-inference-service] best_gid_index="$idx" [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] best_gid_index="3" [e2e-llm-inference-service] echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] [e2e-llm-inference-service] export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] fi [e2e-llm-inference-service] echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] [e2e-llm-inference-service] # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] fi [e2e-llm-inference-service] # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] # Prefill is the KV producer; without a connector here decode has nothing to fetch. [e2e-llm-inference-service] if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\"}'" [e2e-llm-inference-service] echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] --served-model-name "facebook/opt-125m" \ [e2e-llm-inference-service] --port 8000 \ [e2e-llm-inference-service] --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] --enable-ssl-refresh \ [e2e-llm-inference-service] --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] $@" -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:274 # Diagnostics for 'tls-verification-test' in 'e2e-test-llm-tls-resources-e4cad0c3' [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:275 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:278 # LLMInferenceService tls-verification-test [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:287 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:18Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:18Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:18Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-eppconfig-default: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-eppconfig-default-pd: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-sglang-template: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] resourceVersion: '82071' [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-tls-verification-86160844 [e2e-llm-inference-service] - name: workload-pd-cpu-tls-verificatio-cc747c7a [e2e-llm-inference-service] - name: model-fb-opt-125m-tls-verificat-7c5fca28 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-eppconfig-default: kserve-config-llm-scheduler-eppconfig-default [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-eppconfig-default-pd: kserve-config-llm-scheduler-eppconfig-default-pd [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-sglang-template: kserve-config-sglang-template [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:10:12Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:58Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:11:46Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:48Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:10:21Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:10:21Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://ac47b216634cb4a6dbfde57801eb81f0-117499614.us-east-1.elb.amazonaws.com/e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 None e2e-test-llm-tls-resources-e4cad0c3 Normal Scheduled Successfully assigned e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv to ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 multus Normal AddedInterface Add eth0 [10.133.0.74/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-142-209.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-142-209.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-142-209.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-142-209.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-142-209.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-142-209.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:49 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-142-209.ec2.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:49 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-142-209.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:49 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-142-209.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:11:56 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-142-209.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.74:8001/health": dial tcp 10.133.0.74:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-56b598698f-4h4fv [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 None e2e-test-llm-tls-resources-e4cad0c3 Normal Scheduled Successfully assigned e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n to ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-130-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-130-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-130-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:49 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-130-79.ec2.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:49 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-130-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:49 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-130-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:11:36 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-130-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.49:8000/health": dial tcp 10.132.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-prefill-7f8db478bb-9l47n [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-prefill-7f8db478bb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 None e2e-test-llm-tls-resources-e4cad0c3 Normal Scheduled Successfully assigned e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh to ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 multus Normal AddedInterface Add eth0 [10.134.0.67/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-141-13.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-141-13.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:46 e2e-test-llm-tls-resources-e4cad0c3 kubelet/ip-10-0-141-13.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-router-scheduler-78cc9f8576 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-56b598698f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:39 e2e-test-llm-tls-resources-e4cad0c3 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy tls-verification-test-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "tls-verification-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:45 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-tls-resources-e4ca098b05fa5f51c9840356eab55a0cbfac [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:57 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:56 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:09:57 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:80 2026-09-18 23:12:06 e2e-test-llm-tls-resources-e4cad0c3 LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [tls-verification-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:252 ### EndpointSlice tls-verification-test-epp-service-45l8s (service=tls-verification-test-epp-service): [{'addresses': ['10.134.0.67'], 'ready': True, 'targetRef': 'tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh'}] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:252 ### EndpointSlice tls-verification-test-inference-pool-ip-e386aca5-pxl7d (service=tls-verification-test-inference-pool-ip-e386aca5): [{'addresses': ['10.133.0.74'], 'ready': True, 'targetRef': 'tls-verification-test-kserve-56b598698f-4h4fv'}, {'addresses': ['10.132.0.49'], 'ready': True, 'targetRef': 'tls-verification-test-kserve-prefill-7f8db478bb-9l47n'}] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:252 ### EndpointSlice tls-verification-test-kserve-workload-svc-hg4tr (service=tls-verification-test-kserve-workload-svc): [{'addresses': ['10.133.0.74'], 'ready': True, 'targetRef': 'tls-verification-test-kserve-56b598698f-4h4fv'}, {'addresses': ['10.132.0.49'], 'ready': True, 'targetRef': 'tls-verification-test-kserve-prefill-7f8db478bb-9l47n'}] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:172 ### Pod tls-verification-test-kserve-56b598698f-4h4fv (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b'{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.453646716Z","body":"Proxy starting","service.name":"llm-d-router-disagg-sidecar","Built on":"","From Git SHA":"unknown","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.453682461Z","body":"Proxy configuration","service.name":"llm-d-router-disagg-sidecar","config":"{\\"Port\\":\\"8000\\",\\"KVConnector\\":\\"nixlv2\\",\\"ECConnector\\":\\"\\",\\"DataParallelSize\\":1,\\"MaxIdleConnsPerHost\\":1024,\\"EnablePrefillerSampling\\":false,\\"PrefillMaxRetries\\":0,\\"PrefillRetryBackoff\\":200000000,\\"UseTLSForPrefiller\\":true,\\"UseTLSForDecoder\\":true,\\"UseTLSForEncoder\\":false,\\"InsecureSkipVerifyForPrefiller\\":false,\\"InsecureSkipVerifyForEncoder\\":false,\\"InsecureSkipVerifyForDecoder\\":false,\\"SecureServing\\":true,\\"CertPath\\":\\"/var/run/kserve/tls\\",\\"TLSMinVersion\\":0,\\"TLSCipherSuites\\":null,\\"MetricsPort\\":0,\\"MetricsCertDir\\":\\"\\",\\"MooncakeBootstrapPort\\":8998,\\"P2PConnectorPort\\":7777,\\"EnableP2PPull\\":false,\\"EnableSSRFProtection\\":true,\\"InferencePoolNamespace\\":\\"e2e-test-llm-tls-resources-e4cad0c3\\",\\"InferencePoolName\\":\\"tls-verification-test-inference-pool\\",\\"PoolGroup\\":\\"inference.networking.x-k8s.io\\",\\"DecodeChunkSize\\":0,\\"Tracing\\":false,\\"MoRIIOWriteMode\\":false,\\"MoRIIODecodeNotifyPort\\":61005,\\"MoRIIODecodeHandshakePort\\":6301,\\"MoRIIODecodePodIP\\":\\"\\",\\"MoRIIOParallelDispatch\\":false,\\"MoRIIOParallelDecodeWaitTimeout\\":30000000000,\\"MoRIIOPrefillHandshakePort\\":6301,\\"MoRIIOPrefillNotifyPort\\":61005,\\"MoRIIOTPSize\\":1,\\"MoRIIODPSize\\":1,\\"MoRIIORemoteHosts\\":null,\\"MoRIIODPSizeLocal\\":0,\\"MoRIIODecodeHosts\\":null,\\"MoRIIORemoteHostSpecs\\":null,\\"MoRIIODecodeHostSpecs\\":null,\\"MoRIIODecodePodIPSpec\\":\\"\\",\\"DecoderURL\\":\\"https://localhost:8001\\"}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.455187597Z","logger":"allowlist-validator","body":"starting SSRF protection allowlist validator","service.name":"llm-d-router-disagg-sidecar","namespace":"e2e-test-llm-tls-resources-e4cad0c3","poolName":"tls-verification-test-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.468045651Z","logger":"allowlist-validator","body":"InferencePool added","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.477023427Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":0,"targets":{},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.477091792Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":0,"targets":{},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.556331997Z","logger":"allowlist-validator","body":"allowlist validator started successfully","service.name":"llm-d-router-disagg-sidecar","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.558692339Z","logger":"proxy server on port 8000","body":"server TLS configured","service.name":"llm-d-router-disagg-sidecar","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.558709541Z","logger":"proxy server on port 8000","body":"starting","service.name":"llm-d-router-disagg-sidecar","addr":"[::]:8000","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.837676309Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":2,"targets":{"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:47.223718333Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name [e2e-llm-inference-service] ":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:47.855043562Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:49.239213176Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:49.870264781Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:50.245194147Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:50.875584008Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:56.099967495Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:10:16.468995604Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:10:16.479663939Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:10:16.479713578Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:10:46.469581672Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:10:46.488723083Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:10:46.488784861Z","logger":"allowlist [e2e-llm-inference-service] -validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:11:16.470260902Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:11:16.48268816Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:11:16.482753445Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:11:46.128149971Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:11:46.153091499Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:11:46.470639078Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:11:46.481361182Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:11:46.481419888Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:12:06.119275582Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:12:06.155638638Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:12:16.471559553Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:12:16.482715409Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-dis [e2e-llm-inference-service] agg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:12:16.48280882Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:12:46.47249154Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:12:46.485485037Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:12:46.485629465Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:13:16.473514473Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:13:16.484223813Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:13:16.48430444Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:13:46.474000852Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:13:46.483775022Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:13:46.48382904Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:14:16.474561244Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:14:16.494385485Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_n [e2e-llm-inference-service] umber":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:14:16.494452155Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:14:46.474897849Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:14:46.485091934Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:14:46.485150242Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:15:16.475292387Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:15:16.485661492Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:15:16.485722969Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:15:46.475910278Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:15:46.486919636Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:15:46.48699502Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:16:16.476537396Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:16:16.488504052Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:16:16.488590268Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCou [e2e-llm-inference-service] nt":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:16:46.476843675Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:16:46.489247032Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:16:46.489297095Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:17:16.477782013Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:17:16.490319944Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:17:16.4903778Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:17:46.478788661Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:17:46.489057047Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:17:46.489120462Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:18:16.479389085Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:18:16.490591557Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:18:16.49066656Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_ [e2e-llm-inference-service] text":"INFO","timestamp":"2026-09-18T23:18:46.480269075Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:18:46.490733292Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:18:46.490836812Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:19:16.48129953Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:19:16.495015778Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:19:16.495071475Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:19:46.481939827Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:19:46.493100154Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:19:46.493159089Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:20:16.482750181Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:20:16.493400522Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:20:16.493473433Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:20:46.482910138Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-tes [e2e-llm-inference-service] t-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:20:46.492978837Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:20:46.493036172Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:21:16.483526914Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:21:16.493474474Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:21:16.493536835Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:21:46.484513695Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:21:46.494526662Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:21:46.494585022Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:22:16.484912108Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:22:16.496153106Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:22:16.496214182Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:22:46.485463366Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:22:46.49920591Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-rout [e2e-llm-inference-service] er-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:22:46.499271027Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:23:16.486225066Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:23:16.496440548Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:23:16.496513056Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:23:46.486651623Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:23:46.496733192Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:23:46.496783405Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:24:16.487288894Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:24:16.49722827Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:24:16.497295598Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:24:46.487724538Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:24:46.496252435Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"s [e2e-llm-inference-service] everity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:24:46.496298459Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:25:16.488358999Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:25:16.498997697Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:25:16.499060192Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:25:46.488885403Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:25:46.499141206Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:25:46.499199927Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:26:16.489919467Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:26:16.500544146Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:26:16.50060465Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:26:46.490167736Z","logger":"allowlist-validator","body":"InferencePool updated","service.name":"llm-d-router-disagg-sidecar","name":"tls-verification-test-inference-pool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:26:46.499707618Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar","targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:26:46.499759986Z","logger":"allowlist-validator","body":"rebuilt allowlist","service.name":"llm-d-router-disagg-sidecar"," [e2e-llm-inference-service] targetCount":4,"targets":{"10.132.0.49":{},"10.133.0.74":{},"tls-verification-test-kserve-56b598698f-4h4fv":{},"tls-verification-test-kserve-prefill-7f8db478bb-9l47n":{}},"severity_number":9}\n' [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b"2026-09-18 23:09:47.216 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://example-models/facebook/opt-125m', '/mnt/models')]\n2026-09-18 23:09:47.216 1 storage.initializer INFO [kserve_storage.py:download():331] Copying contents of s3://example-models/facebook/opt-125m to local\n2026-09-18 23:09:47.236 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():626] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmp87pqna8p.crt\n2026-09-18 23:09:47.708 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/.gitignore (1 bytes)\n2026-09-18 23:09:47.708 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/.gitattributes.lock\n2026-09-18 23:09:47.708 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/.gitattributes.metadata (101 bytes)\n2026-09-18 23:09:47.708 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/LICENSE.md.lock\n2026-09-18 23:09:47.708 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/LICENSE.md.metadata (101 bytes)\n2026-09-18 23:09:47.708 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/README.md.lock\n2026-09-18 23:09:47.708 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/README.md.metadata (101 bytes)\n2026-09-18 23:09:47.708 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/config.json.lock\n2026-09-18 23:09:47.708 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/config.json.metadata (101 bytes)\n2026-09-18 23:09:47.708 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/generation_config.json.lock\n2026-09-18 23:09:47.708 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/generation_config.json.metadata (101 bytes)\n2026-09-18 23:09:47.708 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/merges.txt.lock\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/merges.txt.metadata (101 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.lock\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.metadata (125 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.lock\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.metadata (101 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.lock\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.metadata (100 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer DEBUG [kserve_ [e2e-llm-inference-service] storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/vocab.json.lock\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/vocab.json.metadata (100 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.gitattributes (1173 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/LICENSE.md (11117 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/README.md (7099 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/config.json (651 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/generation_config.json (137 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/merges.txt (456318 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/pytorch_model.bin (250540281 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/special_tokens_map.json (441 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/tokenizer_config.json (685 bytes)\n2026-09-18 23:09:47.709 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/vocab.json (898822 bytes)\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/.gitignore\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/.gitattributes.metadata\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/README.md.metadata\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/LICENSE.md.metadata\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/generation_config.json.metadata\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/config.json.metadata\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/merges.txt.metadata\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.metadata\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.metadata\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.metadata\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/vocab.json.metadata\n2026-09-18 23:09:48.945 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.gitattributes\n2026-09-18 23:09:48.946 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/README.md\n2026-09-18 23:09:48. [e2e-llm-inference-service] 946 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/LICENSE.md\n2026-09-18 23:09:48.946 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/config.json\n2026-09-18 23:09:48.946 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/generation_config.json\n2026-09-18 23:09:48.946 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/special_tokens_map.json\n2026-09-18 23:09:48.946 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/tokenizer_config.json\n2026-09-18 23:09:48.946 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/merges.txt\n2026-09-18 23:09:48.946 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/vocab.json\n2026-09-18 23:09:48.946 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/pytorch_model.bin\n2026-09-18 23:09:48.947 1 storage.initializer INFO [kserve_storage.py:download():420] Successfully copied s3://example-models/facebook/opt-125m to /mnt/models\n2026-09-18 23:09:48.947 1 storage.initializer INFO [kserve_storage.py:download():421] Model downloaded in 1.7312281750000693 seconds.\n" [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b'(EngineCore pid=76) DEBUG 09-18 23:11:54 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode=\'hf\'\n(EngineCore pid=76) DEBUG 09-18 23:11:55 [utils/gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1\n(EngineCore pid=76) INFO 09-18 23:11:55 [config/vllm.py:790] Asynchronous scheduling is disabled.\n(APIServer pid=1) DEBUG 09-18 23:11:55 [v1/engine/utils.py:1158] READY from local core engine process 0.\n(EngineCore pid=76) WARNING 09-18 23:11:55 [config/vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.\n(EngineCore pid=76) DEBUG 09-18 23:11:55 [v1/engine/core.py:1158] EngineCore waiting for work.\n(APIServer pid=1) DEBUG 09-18 23:11:55 [v1/metrics/loggers.py:273] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 227\n(EngineCore pid=76) DEBUG 09-18 23:11:55 [v1/engine/core.py:1158] EngineCore waiting for work.\n(EngineCore pid=76) DEBUG 09-18 23:11:55 [v1/engine/core.py:1158] EngineCore waiting for work.\n(APIServer pid=1) INFO 09-18 23:11:55 [entrypoints/openai/api_server.py:590] Supported tasks: [\'generate\']\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/base.py:197] Warming up chat template processing...\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] Traceback (most recent call last):\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools)\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] raise ValueError(\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:141] There is no chat template fallback for /mnt/models\n(APIServer pid=1) INFO 09-18 23:11:56 [renderers/hf.py:314] Detected the chat template content format to be \'string\'. You can set `--chat-template-content-format` to override this.\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] Traceback (most recent call last):\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools)\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] raise ValueError(\r\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templ [e2e-llm-inference-service] ates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating\n(APIServer pid=1) DEBUG 09-18 23:11:56 [renderers/base.py:205] This model does not support chat template.\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8001\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:37] Available routes are:\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: HEAD, GET\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /docs, Methods: HEAD, GET\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /redoc, Methods: HEAD, GET\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /detokenize, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /load, Methods: GET\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /version, Methods: GET\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /health, Methods: GET\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /ping, Methods: GET\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /ping, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST\n(APIServer pid=1) INFO: Started server process [1]\n(APIServer pid=1) INFO: Waiting for application startup.\n(APIServer pid=1) INFO 09-18 23:11:56 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: [\'/var/run/kserve/tls/tls.key\', \'/var/run/kserve/tls/tls.crt\']\n(APIServer pid=1) INFO: Application startup complete.\n(APIServer pid=1) DEBUG 09-18 23:12:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:12:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) WARNING 09-18 23:12:10 [tracing/utils.py:72] Receiv [e2e-llm-inference-service] ed a request with trace context but tracing is disabled\n(APIServer pid=1) DEBUG 09-18 23:12:10 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors).\n(EngineCore pid=76) DEBUG 09-18 23:12:10 [v1/engine/core.py:1170] EngineCore loop active.\n(EngineCore pid=76) DEBUG 09-18 23:12:10 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=5, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:10 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cud [e2e-llm-inference-service] agraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3888] Running [e2e-llm-inference-service] batch with cudagraph_mode: NONE, batch_descriptor: BatchDescriptor(num_tokens=1, num_reqs=None, uniform=False, has_lora=False, num_active_loras=0), should_ubatch: False, num_tokens_across_dp: None\n(EngineCore pid=76) DEBUG 09-18 23:12:13 [v1/worker/gpu_model_runner.py:3909] ubatch_slices: None, ubatch_slices_padded: None\n(EngineCore pid=76) DEBUG 09-18 23:12:14 [v1/engine/core.py:1158] EngineCore waiting for work.\n(APIServer pid=1) INFO: 10.132.0.27:0 - "POST /v1/completions HTTP/1.1" 200 OK\n(APIServer pid=1) INFO 09-18 23:12:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.5 tokens/s, Avg generation throughput: 2.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) INFO 09-18 23:12:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:12:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:12:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:12:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 to [e2e-llm-inference-service] kens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:26 [v1/metrics/loggers.py:2 [e2e-llm-inference-service] 59] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix [e2e-llm-inference-service] cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throug [e2e-llm-inference-service] hput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:24:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:24:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:24:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:24:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:24:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:24:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:46 [v1/metrics/ [e2e-llm-inference-service] loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:16 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:26 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:36 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:46 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:56 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:27:06 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n' [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:172 ### Pod tls-verification-test-kserve-prefill-7f8db478bb-9l47n (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b"2026-09-18 23:09:46.770 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('s3://example-models/facebook/opt-125m', '/mnt/models')]\n2026-09-18 23:09:46.771 1 storage.initializer INFO [kserve_storage.py:download():331] Copying contents of s3://example-models/facebook/opt-125m to local\n2026-09-18 23:09:46.789 1 storage.initializer INFO [kserve_storage.py:_configure_global_ca_bundle():626] Global ca bundle file(/etc/ssl/custom-certs/cabundle.crt) combined with the default trust store at /tmp/tmpwmztjcke.crt\n2026-09-18 23:09:47.270 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/.gitignore (1 bytes)\n2026-09-18 23:09:47.270 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/.gitattributes.lock\n2026-09-18 23:09:47.270 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/.gitattributes.metadata (101 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/LICENSE.md.lock\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/LICENSE.md.metadata (101 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/README.md.lock\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/README.md.metadata (101 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/config.json.lock\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/config.json.metadata (101 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/generation_config.json.lock\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/generation_config.json.metadata (101 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/merges.txt.lock\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/merges.txt.metadata (101 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.lock\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.metadata (125 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.lock\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.metadata (101 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer DEBUG [kserve_storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.lock\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.metadata (100 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer DEBUG [kserve_ [e2e-llm-inference-service] storage.py:_download_s3():712] Skipping: facebook/opt-125m/.cache/huggingface/download/vocab.json.lock\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.cache/huggingface/download/vocab.json.metadata (100 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/.gitattributes (1173 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/LICENSE.md (11117 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/README.md (7099 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/config.json (651 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/generation_config.json (137 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/merges.txt (456318 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/pytorch_model.bin (250540281 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/special_tokens_map.json (441 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/tokenizer_config.json (685 bytes)\n2026-09-18 23:09:47.271 1 storage.initializer INFO [kserve_storage.py:_download_s3():715] Found S3 object: facebook/opt-125m/vocab.json (898822 bytes)\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/.gitignore\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/LICENSE.md.metadata\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/.gitattributes.metadata\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/config.json.metadata\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/README.md.metadata\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/generation_config.json.metadata\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/pytorch_model.bin.metadata\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/special_tokens_map.json.metadata\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/tokenizer_config.json.metadata\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/merges.txt.metadata\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.cache/huggingface/download/vocab.json.metadata\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/.gitattributes\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/LICENSE.md\n2026-09-18 23:09:48 [e2e-llm-inference-service] .578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/generation_config.json\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/README.md\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/config.json\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/special_tokens_map.json\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/tokenizer_config.json\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/merges.txt\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/vocab.json\n2026-09-18 23:09:48.578 1 storage.initializer INFO [kserve_storage.py:_download_s3():767] Downloaded object facebook/opt-125m/pytorch_model.bin\n2026-09-18 23:09:48.579 1 storage.initializer INFO [kserve_storage.py:download():420] Successfully copied s3://example-models/facebook/opt-125m to /mnt/models\n2026-09-18 23:09:48.579 1 storage.initializer INFO [kserve_storage.py:download():421] Model downloaded in 1.8089769609996438 seconds.\n" [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b'(APIServer pid=1) DEBUG 09-18 23:12:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:12:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:13:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:13:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:13:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:13:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:13:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:13:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:13:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:13:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:13:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:14:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:14:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:14:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:14:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:14:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:46 [v1/engine/async_llm.py:875] Called c [e2e-llm-inference-service] heck_health.\n(APIServer pid=1) DEBUG 09-18 23:14:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:14:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:14:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:14:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:15:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:15:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:15:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:15:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:15:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:15:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:15:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:15:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:15:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:16:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:16:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:16:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:16:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09 [e2e-llm-inference-service] -18 23:16:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:16:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:16:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:16:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:16:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:17:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:17:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:17:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:17:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:17:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:17:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:17:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:17:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:17:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:18:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:18:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:18:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:18:28 [v1/metrics/loggers.py:259] [e2e-llm-inference-service] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:18:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:18:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:18:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:18:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:18:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:19:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:19:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:19:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:19:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:19:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:19:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:19:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:19:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:19:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:20:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:20:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:20:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 [e2e-llm-inference-service] tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:20:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:20:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:20:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:20:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:20:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:20:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:21:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:21:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:21:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:21:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:21:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:21:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:21:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:21:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:21:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:22:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit ra [e2e-llm-inference-service] te: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:22:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:22:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:22:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:22:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:22:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:22:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:22:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:22:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:23:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:23:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:23:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:23:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:23:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:23:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:23:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:23:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:23:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 [e2e-llm-inference-service] 23:24:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:24:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:24:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:24:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:24:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:24:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:24:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:24:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:24:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:24:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:24:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:24:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:24:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:24:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:25:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:25:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:25:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:25:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:25:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:25:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:25:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:25:56 [v1/engine/async_llm.py:875] Ca [e2e-llm-inference-service] lled check_health.\n(APIServer pid=1) DEBUG 09-18 23:25:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:26:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:26:16 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:26:18 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:26 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:26:28 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:36 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:26:38 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:26:46 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:26:48 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:26:56 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:26:58 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n(APIServer pid=1) DEBUG 09-18 23:27:06 [v1/engine/async_llm.py:875] Called check_health.\n(APIServer pid=1) DEBUG 09-18 23:27:08 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%\n' [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:172 ### Pod tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:226 b'{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.673891062Z","logger":"setup","caller":"runner/runner.go:221","body":"GIE build","service.name":"llm-d-epp","commit-sha":"unknown","build-ref":"","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.674164309Z","logger":"setup","caller":"runner/runner.go:245","body":"Flags processed","service.name":"llm-d-epp","flags":{"allow-experimental-plugins":false,"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\\nkind: EndpointPickerConfig\\nplugins:\\n- type: prefill-filter\\n- type: decode-filter\\n- type: queue-scorer\\n- type: kv-cache-utilization-scorer\\n- type: active-request-scorer\\n- type: prefix-cache-scorer\\n- type: max-score-picker\\n- type: always-disagg-pd-decider\\n- parameters:\\n deciders:\\n prefill: always-disagg-pd-decider\\n type: disagg-profile-handler\\n- parameters:\\n scheme: https\\n type: metrics-data-source\\nschedulingProfiles:\\n- name: prefill\\n plugins:\\n - pluginRef: prefill-filter\\n - pluginRef: prefix-cache-scorer\\n weight: 3\\n - pluginRef: queue-scorer\\n weight: 2\\n - pluginRef: kv-cache-utilization-scorer\\n weight: 2\\n - pluginRef: max-score-picker\\n- name: decode\\n plugins:\\n - pluginRef: decode-filter\\n - pluginRef: active-request-scorer\\n weight: 2\\n - pluginRef: prefix-cache-scorer\\n weight: 3\\n - pluginRef: max-score-picker\\n","disable-endpoint-subset-filter":false,"drain-timeout":30000000000,"emit-endpoint-scores":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"fairness-id-metric-label-limit":1000,"feature-gates":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-cert-dir":"","metrics-client-ca-file":"","metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"plugin-state-staleness-threshold":300000000000,"pool-group":"inference.networking.k8s.io","pool-name":"tls-verification-test-inference-pool","pool-namespace":"e2e-test-llm-tls-resources-e4cad0c3","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tls-cipher-suites":{},"tls-min-version":"","tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.674388876Z","logger":"setup.trace","caller":"tracing/telemetry.go:298","body":"init OTel trace exporter","service.name":"llm-d-epp","type":"otlp","protocol":"grpc","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.675861686Z","caller":"loader/configloader.go:99","body":"Loaded raw configuration","service.name":"llm-d-epp","config":"{Plugins: [{Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: active-request-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\\"deciders\\":{\\"prefill\\":\\"always-disagg-pd-decider\\"}}} {Type: metrics-data-source, Parameters: {\\"scheme\\":\\"https\\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.675908867Z","logger":"setup","caller":"runner/runner.go:757","body":"Data layer: ENABLED","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","ti [e2e-llm-inference-service] mestamp":"2026-09-18T23:09:46.67623962Z","logger":"setup","caller":"runner/runner.go:362","body":"Raw config after phase one","service.name":"llm-d-epp","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"active-request-scorer","parameters":null,"type":"active-request-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"active-request-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]},"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.692172903Z","caller":"disagg/disagg_profile_handler.go:161","body":"No deciders.encode configured, E disaggregation disabled","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.692276739Z","logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:96","body":"Creating new UtilizationDetector","service.name":"llm-d-epp","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","stalenessPolicy":"saturated","headroom":0,"severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.692354284Z","caller":"metrics/factories.go:240","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"vllm","mapping":"Mapping{all specs enabled}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.692436393Z","caller":"metrics/factories.go:240","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"sglang","mapping":"Mapping{disabled: [lora, cacheInfo]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.692483262Z","caller":"metrics/factories.go:240","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"atom","mapping":"Mapping{disabled: [lora]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.692543718Z","caller":"metrics/factories.go:240","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.692619169Z","caller":"metrics/factories.go:240","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.692655116Z","caller":"metrics/factories.go:240","body":"Registered engine mapping","service.name":"llm-d-epp","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.692743328Z","caller":"loader/configloader.go:168","body":"Instantiated all plugins and applied system defaults. Effective raw configuration","service.name":"llm-d-epp","config":"{Plugins: [{Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filt [e2e-llm-inference-service] er} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: active-request-scorer, Type: active-request-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\\"deciders\\":{\\"prefill\\":\\"always-disagg-pd-decider\\"}}} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\\"scheme\\":\\"https\\"}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: passthrough-parser, Type: passthrough-parser} {Name: utilization-detector, Type: utilization-detector} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker} {PluginRef: utilization-detector}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker} {PluginRef: utilization-detector}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: , CrossReplicaSyncerPluginRef: , CrossReplicaSyncInterval: nil, CrossReplicaPublishTimeout: nil}, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}, {PluginRef: passthrough-parser}]}}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.6928516Z","caller":"datalayer/data_graph.go:132","body":"auto-created default producer","service.name":"llm-d-epp","producer":"inflight-load-producer/inflight-load-producer","dataKey":"InFlightLoadDataKey/inflight-load-producer","consumer":"active-request-scorer","severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.692869143Z","caller":"approximateprefix/plugin.go:141","body":"Prefix DataProducer initialized","service.name":"llm-d-epp","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250},"severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.692931034Z","caller":"approximateprefix/plugin.go:167","body":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","service.name":"llm-d-epp","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.6929616Z","caller":"datalayer/data_graph.go:132","body":"auto-created default producer","service.name":"llm-d-epp","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"disagg-profile-handler","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.692978688Z","caller":"datalayer/data_graph.go:132","body":"auto-created default producer","service.name":"llm-d-epp","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.693188725Z","caller":"runner/runner.go:824","body":"loaded configuration from file/text successfully","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09: [e2e-llm-inference-service] 46.693200916Z","logger":"setup","caller":"runner/runner.go:390","body":"EPP config after phase two","service.name":"llm-d-epp","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/decode-filter, utilization-detector/utilization-detector], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/prefill-filter, utilization-detector/utilization-detector], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0x275c3e9b31f0 DataConfig:{Sources:[{Plugin:0x275c3e8719a0 Extractors:[0x275c3eae8d00]}] Syncer: SyncInterval:0s PublishTimeout:0s} FlowControlConfig: ParserRegistry:0x275c3eae9480 PropagatePriority:false}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.709025093Z","logger":"setup","caller":"runner/runner.go:439","body":"Setting pprof handlers","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709051149Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709064709Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/profile","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709070353Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/symbol","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709074524Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/allocs","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709078589Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/threadcreate","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709082537Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/block","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.70908654Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/mutex","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709090456Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/cmdline","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709094917Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/trace","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709100166Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/heap","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709104115Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/pprof/goroutine","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709110457Z","caller":"manager/internal.go:201","body":"Registering metrics http server extra handler","service.name":"llm-d-epp","path":"/debug/plugins/state","severity_number" [e2e-llm-inference-service] :5}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.709114766Z","logger":"setup","caller":"runner/runner.go:460","body":"parsed config","service.name":"llm-d-epp","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/decode-filter, utilization-detector/utilization-detector], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/prefill-filter, utilization-detector/utilization-detector], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.709164056Z","logger":"setup","caller":"datalayer/runtime.go:107","body":"Configuring datalayer runtime","service.name":"llm-d-epp","numSources":1,"severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.709179389Z","logger":"setup","caller":"datalayer/runtime.go:141","body":"Processing source","service.name":"llm-d-epp","source":"metrics-data-source","numExtractors":1,"severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.70919356Z","logger":"setup","caller":"datalayer/runtime.go:185","body":"Source configured","service.name":"llm-d-epp","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"],"severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.709213074Z","logger":"setup","caller":"datalayer/runtime.go:248","body":"Datalayer runtime configured","service.name":"llm-d-epp","pollers":1,"notifiers":0,"endpointSources":1,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.709228442Z","logger":"setup","caller":"runner/runner.go:955","body":"Flow Control layer is disabled via the flowControl feature gate, using legacy admission control","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.709336435Z","logger":"setup","caller":"runner/runner.go:326","body":"Controller manager starting","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.709372904Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","body":"Starting metrics server","service.name":"llm-d-epp","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.709409991Z","caller":"runnable/grpc.go:37","body":"gRPC server starting","service.name":"llm-d-epp","name":"health","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.709803828Z","caller":"controller/controller.go:369","body":"Starting EventSource","service.name":"llm-d-epp","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.709844366Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","body":"Serving metrics server","service.name":"llm-d-epp","bindAddress":":9090","secure":false,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.709945053Z","caller":"runnable/grpc.go:43","body":"gRPC server listening","service.name":"llm-d-epp","name":"health","port":9003,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.710291006Z","caller":"controller/controller.go:369","body":"Starting EventSource","service.name":"llm-d-epp","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.710333111Z","caller":"controller/controller.go:369","body":"Starting EventSource","service.name":"llm-d-epp","controller":"pod","c [e2e-llm-inference-service] ontrollerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.710242715Z","caller":"controller/controller.go:369","body":"Starting EventSource","service.name":"llm-d-epp","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.711144128Z","caller":"runnable/grpc.go:37","body":"gRPC server starting","service.name":"llm-d-epp","name":"ext-proc","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.711244017Z","caller":"runnable/grpc.go:43","body":"gRPC server listening","service.name":"llm-d-epp","name":"ext-proc","port":9002,"severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.714111712Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:507","body":"Caches populated","service.name":"llm-d-epp","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.36.4/tools/cache/reflector.go:343","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.714247669Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:507","body":"Caches populated","service.name":"llm-d-epp","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.36.4/tools/cache/reflector.go:343","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.714548653Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:507","body":"Caches populated","service.name":"llm-d-epp","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.36.4/tools/cache/reflector.go:343","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.715054723Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:507","body":"Caches populated","service.name":"llm-d-epp","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.36.4/tools/cache/reflector.go:343","severity_number":5}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.91160862Z","caller":"controller/controller.go:302","body":"Starting Controller","service.name":"llm-d-epp","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.911638452Z","caller":"controller/controller.go:302","body":"Starting Controller","service.name":"llm-d-epp","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.911662058Z","caller":"controller/controller.go:305","body":"Starting workers","service.name":"llm-d-epp","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.911667943Z","caller":"controller/controller.go:305","body":"Starting workers","service.name":"llm-d-epp","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.911608653Z","caller":"controller/controller.go:302","body":"Starting Controller","service.name":"llm-d-epp","controller":"pod","controllerGroup":"","controllerKind":"Pod","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.911702184Z","caller":"controller/controller.go:305","body":"Starting workers","service.name":"llm-d-epp","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1,"severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.911751068Z","caller":"controller/controller.go:302","body":"Starting Controller","service.name":"llm-d-epp","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai"," [e2e-llm-inference-service] controllerKind":"InferenceModelRewrite","severity_number":9}\n{"severity_text":"INFO","timestamp":"2026-09-18T23:09:46.91176597Z","caller":"controller/controller.go:305","body":"Starting workers","service.name":"llm-d-epp","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1,"severity_number":9}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:46.911797911Z","caller":"controller/inferencepool_reconciler.go:49","body":"Reconciling InferencePool","service.name":"llm-d-epp","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"tls-verification-test-inference-pool","namespace":"e2e-test-llm-tls-resources-e4cad0c3"},"namespace":"e2e-test-llm-tls-resources-e4cad0c3","name":"tls-verification-test-inference-pool","reconcileID":"db8bef60-57d9-49d6-adb7-d697bbaf4700","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:09:57.183449739Z","caller":"controller/inferencepool_reconciler.go:49","body":"Reconciling InferencePool","service.name":"llm-d-epp","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"tls-verification-test-inference-pool","namespace":"e2e-test-llm-tls-resources-e4cad0c3"},"namespace":"e2e-test-llm-tls-resources-e4cad0c3","name":"tls-verification-test-inference-pool","reconcileID":"0ab51bd7-456b-4b19-92d3-825aa1ca23f3","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:11:46.151622507Z","caller":"inflightload/producer.go:381","body":"Injected dynamic attribute into endpoint","service.name":"llm-d-epp","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7f8db478bb-9l47n-rank-0","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:11:46.151709906Z","caller":"datastore/datastore.go:373","body":"Pod added","service.name":"llm-d-epp","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"tls-verification-test-kserve-prefill-7f8db478bb-9l47n","namespace":"e2e-test-llm-tls-resources-e4cad0c3"},"namespace":"e2e-test-llm-tls-resources-e4cad0c3","name":"tls-verification-test-kserve-prefill-7f8db478bb-9l47n","reconcileID":"c1d24fcf-2f4b-426e-a53f-1fcd31a716b4","name":"tls-verification-test-kserve-prefill-7f8db478bb-9l47n","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:12:06.153600479Z","caller":"inflightload/producer.go:381","body":"Injected dynamic attribute into endpoint","service.name":"llm-d-epp","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv-rank-0","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:12:06.153753754Z","caller":"datastore/datastore.go:373","body":"Pod added","service.name":"llm-d-epp","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"tls-verification-test-kserve-56b598698f-4h4fv","namespace":"e2e-test-llm-tls-resources-e4cad0c3"},"namespace":"e2e-test-llm-tls-resources-e4cad0c3","name":"tls-verification-test-kserve-56b598698f-4h4fv","reconcileID":"ddea2050-c5f4-4ff6-86cc-d8c0512bb71f","name":"tls-verification-test-kserve-56b598698f-4h4fv","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:12:07.508873598Z","caller":"handlers/server.go:493","body":"EPP received request","service.name":"llm-d-epp","x-request-id":"8ec8e3a4-57ee-472a-8673-b0732998f701","trace_id":"eab07411d976371275cc6941933304a8","span_id":"fbe4d0148c840bfc","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:12:07.509611363Z","caller":"handlers/server.go:783","body":"EPP sent request body response(s) to proxy","service.name":"llm-d-epp","x-request-id":"8ec8e3a4-57ee-472a-8673-b0732998f701","trace_id":"eab07411d976371275cc6941933304a8","span_id":"fbe4d0148c840bfc","modelName":"facebook/opt-125m","targetModelName":"facebo [e2e-llm-inference-service] ok/opt-125m","severity_number":5}\n{"severity_text":"DEBUG","timestamp":"2026-09-18T23:12:14.016062948Z","caller":"handlers/server.go:806","body":"EPP sent response body back to proxy","service.name":"llm-d-epp","x-request-id":"8ec8e3a4-57ee-472a-8673-b0732998f701","trace_id":"eab07411d976371275cc6941933304a8","span_id":"fbe4d0148c840bfc","severity_number":5}\n' [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 601b0594-fae6-4d72-bed2-5411a4771a7d [e2e-llm-inference-service] resourceVersion: '80893' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.67 [e2e-llm-inference-service] nodeName: ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh [e2e-llm-inference-service] uid: 5794c3e9-efb3-451c-9257-94778463c017 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: bbf95253-48d2-4d8a-8099-7ead0755abe6 [e2e-llm-inference-service] resourceVersion: '82058' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.49 [e2e-llm-inference-service] nodeName: ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-7f8db478bb-9l47n [e2e-llm-inference-service] uid: 01e70533-b9ac-4b8d-a073-838c050337d9 [e2e-llm-inference-service] - ip: 10.133.0.74 [e2e-llm-inference-service] nodeName: ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-56b598698f-4h4fv [e2e-llm-inference-service] uid: 5318e729-a528-4255-87ad-f1080b6ffe7d [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-56b598698f-4h4fv [e2e-llm-inference-service] generateName: tls-verification-test-kserve-56b598698f- [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 5318e729-a528-4255-87ad-f1080b6ffe7d [e2e-llm-inference-service] resourceVersion: '82057' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 56b598698f [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.74/23"],"mac_address":"0a:58:0a:85:00:4a","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.74/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.74\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:4a\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: tls-verification-test-kserve-56b598698f [e2e-llm-inference-service] uid: d45b1791-1928-4dcb-8f0e-02d7bb663686 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-142-209 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d45b1791-1928-4dcb-8f0e-02d7bb663686"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.74"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-mjnfx [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: tls-verification-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-mjnfx [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001060000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] - name: kube-api-access-mjnfx [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001060000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n if\ [e2e-llm-inference-service] \ [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\")\ [e2e-llm-inference-service] \ | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n \ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n fi\n\ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n \ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep -H\ [e2e-llm-inference-service] \ . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n# ODH\ [e2e-llm-inference-service] \ supports older accelerator images, so only pass it when available.\n\n\n#\ [e2e-llm-inference-service] \ --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=\"\ [e2e-llm-inference-service] \"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\\ [e2e-llm-inference-service] n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head -1)\" = \"0.18.0\" ]; then\n\ [e2e-llm-inference-service] \ SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\nfi\n\n# A user-supplied\ [e2e-llm-inference-service] \ --kv-transfer-config always wins; KServe only fills the flag when it is unset.\n\ [e2e-llm-inference-service] KV_TRANSFER_ARGS=\"\"\nif [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv_transfer_config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv-transfer-config\"* ]] && [[ \"$*\" != *\"--kv_transfer_config\"\ [e2e-llm-inference-service] * ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\n if [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\\ [e2e-llm-inference-service] .[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.22.0\" ]; then\n KV_TRANSFER_ARGS=\"\"\n fi\n #\ [e2e-llm-inference-service] \ This template is only composed for a disaggregated P/D topology (spec.prefill\ [e2e-llm-inference-service] \ set).\n # Decode is the KV consumer; without a connector here it recomputes\ [e2e-llm-inference-service] \ prefill's KV.\n if [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n # Only inject\ [e2e-llm-inference-service] \ when NIXL is importable. CPU and other non-NIXL engine images\n # raise\ [e2e-llm-inference-service] \ RuntimeError(\"NIXL is not available\") and never finish starting.\n NIXL_PY=$(command\ [e2e-llm-inference-service] \ -v python3 || command -v python || true)\n if [ -n \"${NIXL_PY}\" ] &&\ [e2e-llm-inference-service] \ \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] --kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\",\\\"kv_role\\\ [e2e-llm-inference-service] \":\\\"kv_consumer\\\"}'\"\n echo \"[kv-transfer] NIXL available, enabling\ [e2e-llm-inference-service] \ NixlConnector (kv_consumer)\"\n else\n echo \"[kv-transfer] NIXL not\ [e2e-llm-inference-service] \ available, P/D KV transfer stays disabled\"\n fi\n fi\nfi\n\neval \"exec\ [e2e-llm-inference-service] \ vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\" \"\ [e2e-llm-inference-service] publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m\" \\\ [e2e-llm-inference-service] \n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-mjnfx [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001060000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-kserve [e2e-llm-inference-service] serviceAccount: tls-verification-test-kserve [e2e-llm-inference-service] nodeName: ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c33,c2 [e2e-llm-inference-service] fsGroup: 1001060000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-pdcfl [e2e-llm-inference-service] - name: tls-verification-test-kserve-dockercfg-jfflg [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:49Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] hostIP: 10.0.142.209 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.142.209 [e2e-llm-inference-service] podIP: 10.133.0.74 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.74 [e2e-llm-inference-service] startTime: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar@sha256:8f710adc7cde2ba7289e68fdabd2c271737247727b0c8a9bdf00ad44bd09b34f [e2e-llm-inference-service] containerID: cri-o://915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-mjnfx [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001060000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001060000 [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] finishedAt: '2026-09-18T23:09:49Z' [e2e-llm-inference-service] containerID: cri-o://edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] containerID: cri-o://edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-mjnfx [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001060000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001060000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-09-18T23:09:49Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-mjnfx [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001060000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001060000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-7f8db478bb-9l47n [e2e-llm-inference-service] generateName: tls-verification-test-kserve-prefill-7f8db478bb- [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 01e70533-b9ac-4b8d-a073-838c050337d9 [e2e-llm-inference-service] resourceVersion: '81845' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7f8db478bb [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.49/23"],"mac_address":"0a:58:0a:84:00:31","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.49/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.49\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:31\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-7f8db478bb [e2e-llm-inference-service] uid: 7a6dc14c-6ea0-4d1e-b432-953ce1498ce3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-130-79 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7a6dc14c-6ea0-4d1e-b432-953ce1498ce3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:11:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.49"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-z8kq8 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] - name: kube-api-access-z8kq8 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001060000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n if\ [e2e-llm-inference-service] \ [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\")\ [e2e-llm-inference-service] \ | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n \ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n fi\n\ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n \ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep -H\ [e2e-llm-inference-service] \ . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n# ODH\ [e2e-llm-inference-service] \ supports older accelerator images, so only pass it when available.\n\n\n#\ [e2e-llm-inference-service] \ --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=\"\ [e2e-llm-inference-service] \"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\\ [e2e-llm-inference-service] n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head -1)\" = \"0.18.0\" ]; then\n\ [e2e-llm-inference-service] \ SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\nfi\n\n# A user-supplied\ [e2e-llm-inference-service] \ --kv-transfer-config always wins; KServe only fills the flag when it is unset.\n\ [e2e-llm-inference-service] KV_TRANSFER_ARGS=\"\"\nif [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv_transfer_config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv-transfer-config\"* ]] && [[ \"$*\" != *\"--kv_transfer_config\"\ [e2e-llm-inference-service] * ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\n if [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\\ [e2e-llm-inference-service] .[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.22.0\" ]; then\n KV_TRANSFER_ARGS=\"\"\n fi\n #\ [e2e-llm-inference-service] \ This template is only composed for a disaggregated P/D topology (spec.prefill\ [e2e-llm-inference-service] \ set).\n # Prefill is the KV producer; without a connector here decode has\ [e2e-llm-inference-service] \ nothing to fetch.\n if [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n # Only inject\ [e2e-llm-inference-service] \ when NIXL is importable. CPU and other non-NIXL engine images\n # raise\ [e2e-llm-inference-service] \ RuntimeError(\"NIXL is not available\") and never finish starting.\n NIXL_PY=$(command\ [e2e-llm-inference-service] \ -v python3 || command -v python || true)\n if [ -n \"${NIXL_PY}\" ] &&\ [e2e-llm-inference-service] \ \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] --kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\",\\\"kv_role\\\ [e2e-llm-inference-service] \":\\\"kv_producer\\\"}'\"\n echo \"[kv-transfer] NIXL available, enabling\ [e2e-llm-inference-service] \ NixlConnector (kv_producer)\"\n else\n echo \"[kv-transfer] NIXL not\ [e2e-llm-inference-service] \ available, P/D KV transfer stays disabled\"\n fi\n fi\nfi\n\neval \"exec\ [e2e-llm-inference-service] \ vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\" \\\n\ [e2e-llm-inference-service] \ --port 8000 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-z8kq8 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001060000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c33,c2 [e2e-llm-inference-service] fsGroup: 1001060000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-pdcfl [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:47Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:49Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:11:46Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:11:46Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] hostIP: 10.0.130.79 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.130.79 [e2e-llm-inference-service] podIP: 10.132.0.49 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.49 [e2e-llm-inference-service] startTime: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] finishedAt: '2026-09-18T23:09:48Z' [e2e-llm-inference-service] containerID: cri-o://9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] containerID: cri-o://9e0f1b0644ebd1c27906ecc3eeea31ad584ee20dc379355f5c0f4c74784793be [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-z8kq8 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001060000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001060000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-09-18T23:09:49Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://fc76f5d061af845a6e6a701aa2d54634e2a462e02606f1340d55ca7de1480fda [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-z8kq8 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001060000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001060000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh [e2e-llm-inference-service] generateName: tls-verification-test-kserve-router-scheduler-78cc9f8576- [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 5794c3e9-efb3-451c-9257-94778463c017 [e2e-llm-inference-service] resourceVersion: '80892' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 78cc9f8576 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.67/23"],"mac_address":"0a:58:0a:86:00:43","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.67/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.67\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:43\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-78cc9f8576 [e2e-llm-inference-service] uid: a3e67d18-56d7-467a-87e3-ba1bfba3313a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-13 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a3e67d18-56d7-467a-87e3-ba1bfba3313a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.67"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-vlht4 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - tls-verification-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: active-request-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\n- parameters:\n\ [e2e-llm-inference-service] \ scheme: https\n type: metrics-data-source\nschedulingProfiles:\n- name:\ [e2e-llm-inference-service] \ prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-vlht4 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1001060000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-epp-sa [e2e-llm-inference-service] serviceAccount: tls-verification-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c33,c2 [e2e-llm-inference-service] fsGroup: 1001060000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-pdcfl [e2e-llm-inference-service] - name: tls-verification-test-epp-sa-dockercfg-ng865 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] hostIP: 10.0.141.13 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.13 [e2e-llm-inference-service] podIP: 10.134.0.67 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.67 [e2e-llm-inference-service] startTime: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:c12f30f05219c4138421584f2781f32dda09373ef168b71f854861f529c0792e [e2e-llm-inference-service] containerID: cri-o://247e3176fc401b7f293b0a9dafa0a1d454fc5dee24d63b2866cc1dfc644a60cd [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-vlht4 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1001060000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1001060000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 1f943dea-e5ed-4286-97a0-cdffd2fc552d [e2e-llm-inference-service] resourceVersion: '80135' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: tls-verification-test-epp-sa-dockercfg-ng865 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"tls-verification-test-epp-sa-dockercfg-ng865"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-pdcfl"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-pdcfl [e2e-llm-inference-service] - name: tls-verification-test-epp-sa-dockercfg-ng865 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-pdcfl [e2e-llm-inference-service] - name: tls-verification-test-epp-sa-dockercfg-ng865 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: ec984da1-13e9-485d-8648-e8b011c2029e [e2e-llm-inference-service] resourceVersion: '80098' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: tls-verification-test-kserve-dockercfg-jfflg [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"tls-verification-test-kserve-dockercfg-jfflg"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-pdcfl"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-pdcfl [e2e-llm-inference-service] - name: tls-verification-test-kserve-dockercfg-jfflg [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-pdcfl [e2e-llm-inference-service] - name: tls-verification-test-kserve-dockercfg-jfflg [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 0861e64a-148e-4ce3-8a3b-b29c9c41636f [e2e-llm-inference-service] resourceVersion: '80159' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.225.88 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.225.88 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 162f2b86-356f-485f-9567-16777337019f [e2e-llm-inference-service] resourceVersion: '80123' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.174.230 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.174.230 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 17422144-7153-4933-b9a1-edc4a46376a0 [e2e-llm-inference-service] resourceVersion: '82063' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: tls-verification-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n \ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\"\ [e2e-llm-inference-service] ) | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n\ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n \ [e2e-llm-inference-service] \ fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n\ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep\ [e2e-llm-inference-service] \ -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n#\ [e2e-llm-inference-service] \ ODH supports older accelerator images, so only pass it when available.\n\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe\ [e2e-llm-inference-service] \ only fills the flag when it is unset.\nKV_TRANSFER_ARGS=\"\"\nif [[ \"\ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n # --kv-transfer-config\ [e2e-llm-inference-service] \ with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n\ [e2e-llm-inference-service] \ if [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\\ [e2e-llm-inference-service] n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\" | sort -V | head -1)\" = \"0.22.0\"\ [e2e-llm-inference-service] \ ]; then\n KV_TRANSFER_ARGS=\"\"\n fi\n # This template is only composed\ [e2e-llm-inference-service] \ for a disaggregated P/D topology (spec.prefill set).\n # Decode is the\ [e2e-llm-inference-service] \ KV consumer; without a connector here it recomputes prefill's KV.\n if\ [e2e-llm-inference-service] \ [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n # Only inject when NIXL is importable.\ [e2e-llm-inference-service] \ CPU and other non-NIXL engine images\n # raise RuntimeError(\"NIXL\ [e2e-llm-inference-service] \ is not available\") and never finish starting.\n NIXL_PY=$(command\ [e2e-llm-inference-service] \ -v python3 || command -v python || true)\n if [ -n \"${NIXL_PY}\" ]\ [e2e-llm-inference-service] \ && \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] --kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\",\\\"kv_role\\\ [e2e-llm-inference-service] \":\\\"kv_consumer\\\"}'\"\n echo \"[kv-transfer] NIXL available, enabling\ [e2e-llm-inference-service] \ NixlConnector (kv_consumer)\"\n else\n echo \"[kv-transfer] NIXL\ [e2e-llm-inference-service] \ not available, P/D KV transfer stays disabled\"\n fi\n fi\nfi\n\n\ [e2e-llm-inference-service] eval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-kserve [e2e-llm-inference-service] serviceAccount: tls-verification-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "tls-verification-test-kserve-56b598698f" has successfully [e2e-llm-inference-service] progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 581e5822-455c-4513-99ae-a90c5d383227 [e2e-llm-inference-service] resourceVersion: '81851' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:11:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n \ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\"\ [e2e-llm-inference-service] ) | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n\ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n \ [e2e-llm-inference-service] \ fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n\ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep\ [e2e-llm-inference-service] \ -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n#\ [e2e-llm-inference-service] \ ODH supports older accelerator images, so only pass it when available.\n\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe\ [e2e-llm-inference-service] \ only fills the flag when it is unset.\nKV_TRANSFER_ARGS=\"\"\nif [[ \"\ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n # --kv-transfer-config\ [e2e-llm-inference-service] \ with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n\ [e2e-llm-inference-service] \ if [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\\ [e2e-llm-inference-service] n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\" | sort -V | head -1)\" = \"0.22.0\"\ [e2e-llm-inference-service] \ ]; then\n KV_TRANSFER_ARGS=\"\"\n fi\n # This template is only composed\ [e2e-llm-inference-service] \ for a disaggregated P/D topology (spec.prefill set).\n # Prefill is the\ [e2e-llm-inference-service] \ KV producer; without a connector here decode has nothing to fetch.\n \ [e2e-llm-inference-service] \ if [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n # Only inject when NIXL is\ [e2e-llm-inference-service] \ importable. CPU and other non-NIXL engine images\n # raise RuntimeError(\"\ [e2e-llm-inference-service] NIXL is not available\") and never finish starting.\n NIXL_PY=$(command\ [e2e-llm-inference-service] \ -v python3 || command -v python || true)\n if [ -n \"${NIXL_PY}\" ]\ [e2e-llm-inference-service] \ && \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] --kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\",\\\"kv_role\\\ [e2e-llm-inference-service] \":\\\"kv_producer\\\"}'\"\n echo \"[kv-transfer] NIXL available, enabling\ [e2e-llm-inference-service] \ NixlConnector (kv_producer)\"\n else\n echo \"[kv-transfer] NIXL\ [e2e-llm-inference-service] \ not available, P/D KV transfer stays disabled\"\n fi\n fi\nfi\n\n\ [e2e-llm-inference-service] eval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T23:11:46Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:11:46Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T23:11:46Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "tls-verification-test-kserve-prefill-7f8db478bb" has successfully [e2e-llm-inference-service] progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: b1d1c525-8a90-4383-8f46-354dfcacccb0 [e2e-llm-inference-service] resourceVersion: '80896' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - tls-verification-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n- type: active-request-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n\ [e2e-llm-inference-service] \ deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-epp-sa [e2e-llm-inference-service] serviceAccount: tls-verification-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] lastTransitionTime: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "tls-verification-test-kserve-router-scheduler-78cc9f8576" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-56b598698f [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: d45b1791-1928-4dcb-8f0e-02d7bb663686 [e2e-llm-inference-service] resourceVersion: '82062' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 56b598698f [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: tls-verification-test-kserve [e2e-llm-inference-service] uid: 17422144-7153-4933-b9a1-edc4a46376a0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"17422144-7153-4933-b9a1-edc4a46376a0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:llm-d.ai/routing-sidecar-version: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 56b598698f [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 56b598698f [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] llm-d.ai/routing-sidecar-version: 0.9.0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --model-server-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: tls-verification-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n \ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\"\ [e2e-llm-inference-service] ) | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n\ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n \ [e2e-llm-inference-service] \ fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n\ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep\ [e2e-llm-inference-service] \ -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n#\ [e2e-llm-inference-service] \ ODH supports older accelerator images, so only pass it when available.\n\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe\ [e2e-llm-inference-service] \ only fills the flag when it is unset.\nKV_TRANSFER_ARGS=\"\"\nif [[ \"\ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n # --kv-transfer-config\ [e2e-llm-inference-service] \ with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n\ [e2e-llm-inference-service] \ if [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\\ [e2e-llm-inference-service] n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\" | sort -V | head -1)\" = \"0.22.0\"\ [e2e-llm-inference-service] \ ]; then\n KV_TRANSFER_ARGS=\"\"\n fi\n # This template is only composed\ [e2e-llm-inference-service] \ for a disaggregated P/D topology (spec.prefill set).\n # Decode is the\ [e2e-llm-inference-service] \ KV consumer; without a connector here it recomputes prefill's KV.\n if\ [e2e-llm-inference-service] \ [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n # Only inject when NIXL is importable.\ [e2e-llm-inference-service] \ CPU and other non-NIXL engine images\n # raise RuntimeError(\"NIXL\ [e2e-llm-inference-service] \ is not available\") and never finish starting.\n NIXL_PY=$(command\ [e2e-llm-inference-service] \ -v python3 || command -v python || true)\n if [ -n \"${NIXL_PY}\" ]\ [e2e-llm-inference-service] \ && \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] --kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\",\\\"kv_role\\\ [e2e-llm-inference-service] \":\\\"kv_consumer\\\"}'\"\n echo \"[kv-transfer] NIXL available, enabling\ [e2e-llm-inference-service] \ NixlConnector (kv_consumer)\"\n else\n echo \"[kv-transfer] NIXL\ [e2e-llm-inference-service] \ not available, P/D KV transfer stays disabled\"\n fi\n fi\nfi\n\n\ [e2e-llm-inference-service] eval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-kserve [e2e-llm-inference-service] serviceAccount: tls-verification-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-7f8db478bb [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 7a6dc14c-6ea0-4d1e-b432-953ce1498ce3 [e2e-llm-inference-service] resourceVersion: '81850' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7f8db478bb [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill [e2e-llm-inference-service] uid: 581e5822-455c-4513-99ae-a90c5d383227 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"581e5822-455c-4513-99ae-a90c5d383227"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_NIXL_SIDE_CHANNEL_HOST"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TOKIO_WORKER_THREADS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:11:46Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7f8db478bb [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7f8db478bb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6f4e83a3a4d8420227e6a3cb149b59c0d3b836ec6312029307a62e0720421c78 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - s3://example-models/facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: TOKIO_WORKER_THREADS [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "# Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre\ [e2e-llm-inference-service] \ ]; then\n ARCH=\"$(arch)\"\n case \"${ARCH}\" in\n ppc64le)\n \ [e2e-llm-inference-service] \ if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n\ [e2e-llm-inference-service] \ if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] \ fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n\ [e2e-llm-inference-service] \ if [ -n \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}\" ] && [ -r \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ ]; then\n if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = \"0x0\"\ [e2e-llm-inference-service] ) | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' \"${AIU_AUTOGEN_SENLIB_CONFIG_FILE}\"\ [e2e-llm-inference-service] \ > \"$HOME/.senlib.json\" && [ -s \"$HOME/.senlib.json\" ]; then\n \ [e2e-llm-inference-service] \ export SENLIB_DEVEL_CONFIG_FILE=\"$HOME/.senlib.json\"\n else\n\ [e2e-llm-inference-service] \ echo \"WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE},\ [e2e-llm-inference-service] \ skipping SENLIB config generation\"\n rm -f \"$HOME/.senlib.json\"\ [e2e-llm-inference-service] \n fi\n else\n echo \"WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE\ [e2e-llm-inference-service] \ is not set or not readable, skipping SENLIB config generation\"\n \ [e2e-llm-inference-service] \ fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n\ [e2e-llm-inference-service] \ export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n\ [e2e-llm-inference-service] \ ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n\ [e2e-llm-inference-service] \ source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ \"$KSERVE_INFER_ROCE\"\ [e2e-llm-inference-service] \ = \"true\" ]; then\n echo \"Trying to infer RoCE configs ... \"\n grep\ [e2e-llm-inference-service] \ -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n#\ [e2e-llm-inference-service] \ ODH supports older accelerator images, so only pass it when available.\n\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe\ [e2e-llm-inference-service] \ only fills the flag when it is unset.\nKV_TRANSFER_ARGS=\"\"\nif [[ \"\ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS:-}\" != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n # --kv-transfer-config\ [e2e-llm-inference-service] \ with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n\ [e2e-llm-inference-service] \ if [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\\ [e2e-llm-inference-service] n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\" | sort -V | head -1)\" = \"0.22.0\"\ [e2e-llm-inference-service] \ ]; then\n KV_TRANSFER_ARGS=\"\"\n fi\n # This template is only composed\ [e2e-llm-inference-service] \ for a disaggregated P/D topology (spec.prefill set).\n # Prefill is the\ [e2e-llm-inference-service] \ KV producer; without a connector here decode has nothing to fetch.\n \ [e2e-llm-inference-service] \ if [ -z \"${KV_TRANSFER_ARGS}\" ]; then\n # Only inject when NIXL is\ [e2e-llm-inference-service] \ importable. CPU and other non-NIXL engine images\n # raise RuntimeError(\"\ [e2e-llm-inference-service] NIXL is not available\") and never finish starting.\n NIXL_PY=$(command\ [e2e-llm-inference-service] \ -v python3 || command -v python || true)\n if [ -n \"${NIXL_PY}\" ]\ [e2e-llm-inference-service] \ && \"${NIXL_PY}\" -c \"import nixl\" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] --kv-transfer-config '{\\\"kv_connector\\\":\\\"NixlConnector\\\",\\\"kv_role\\\ [e2e-llm-inference-service] \":\\\"kv_producer\\\"}'\"\n echo \"[kv-transfer] NIXL available, enabling\ [e2e-llm-inference-service] \ NixlConnector (kv_producer)\"\n else\n echo \"[kv-transfer] NIXL\ [e2e-llm-inference-service] \ not available, P/D KV transfer stays disabled\"\n fi\n fi\nfi\n\n\ [e2e-llm-inference-service] eval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] - name: VLLM_NIXL_SIDE_CHANNEL_HOST [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: status.podIP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-78cc9f8576 [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: a3e67d18-56d7-467a-87e3-ba1bfba3313a [e2e-llm-inference-service] resourceVersion: '80895' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 78cc9f8576 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler [e2e-llm-inference-service] uid: b1d1c525-8a90-4383-8f46-354dfcacccb0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b1d1c525-8a90-4383-8f46-354dfcacccb0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 78cc9f8576 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 78cc9f8576 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.11.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - tls-verification-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - '' [e2e-llm-inference-service] - '' [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n- type: active-request-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n\ [e2e-llm-inference-service] \ deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: tls-verification-test-epp-sa [e2e-llm-inference-service] serviceAccount: tls-verification-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-prometheus-scraping [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 0e8a59fa-b9cb-4333-9767-d40ffa65bcb2 [e2e-llm-inference-service] resourceVersion: '80324' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llm-monitoring [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: networking.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:ingress: {} [e2e-llm-inference-service] f:podSelector: {} [e2e-llm-inference-service] f:policyTypes: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] podSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ingress: [e2e-llm-inference-service] - ports: [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] from: [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: openshift-monitoring [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: openshift-user-workload-monitoring [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: redhat-ods-monitoring [e2e-llm-inference-service] - ports: [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] - protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] from: [e2e-llm-inference-service] - namespaceSelector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] kubernetes.io/metadata.name: openshift-ingress [e2e-llm-inference-service] - from: [e2e-llm-inference-service] - podSelector: {} [e2e-llm-inference-service] policyTypes: [e2e-llm-inference-service] - Ingress [e2e-llm-inference-service] apiVersion: networking.k8s.io/v1 [e2e-llm-inference-service] kind: NetworkPolicy [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 040119ad-bd66-4cb1-be20-35f1d67e03b8 [e2e-llm-inference-service] resourceVersion: '80153' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: tls-verification-test-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: tls-verification-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 365a6c76-d6d1-4ddf-a396-70633e1e6534 [e2e-llm-inference-service] resourceVersion: '80104' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: tls-verification-test-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: tls-verification-test-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 27091116-f7ff-44d1-a55c-f16dd2ec9c16 [e2e-llm-inference-service] resourceVersion: '80148' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - patch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - events [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: f4a39702-f18b-44d2-9a3b-63e67841ff6a [e2e-llm-inference-service] resourceVersion: '80102' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-service-45l8s [e2e-llm-inference-service] generateName: tls-verification-test-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: b1b9441c-f215-4b01-b89e-172f2f5c76d6 [e2e-llm-inference-service] resourceVersion: '80894' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: tls-verification-test-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-epp-service [e2e-llm-inference-service] uid: 0861e64a-148e-4ce3-8a3b-b29c9c41636f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T23:10:17Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0861e64a-148e-4ce3-8a3b-b29c9c41636f"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.67 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh [e2e-llm-inference-service] uid: 5794c3e9-efb3-451c-9257-94778463c017 [e2e-llm-inference-service] nodeName: ip-10-0-141-13.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc-hg4tr [e2e-llm-inference-service] generateName: tls-verification-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 0ce9ab32-74bf-4e4f-8638-33b516fb39fa [e2e-llm-inference-service] resourceVersion: '82060' [e2e-llm-inference-service] generation: 5 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] uid: 162f2b86-356f-485f-9567-16777337019f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T23:12:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"162f2b86-356f-485f-9567-16777337019f"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.74 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-56b598698f-4h4fv [e2e-llm-inference-service] uid: 5318e729-a528-4255-87ad-f1080b6ffe7d [e2e-llm-inference-service] nodeName: ip-10-0-142-209.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.49 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-7f8db478bb-9l47n [e2e-llm-inference-service] uid: 01e70533-b9ac-4b8d-a073-838c050337d9 [e2e-llm-inference-service] nodeName: ip-10-0-130-79.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 040119ad-bd66-4cb1-be20-35f1d67e03b8 [e2e-llm-inference-service] resourceVersion: '80153' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-tls-resources-e4cad0c3:tls-verification-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 365a6c76-d6d1-4ddf-a396-70633e1e6534 [e2e-llm-inference-service] resourceVersion: '80104' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-tls-resources-e4cad0c3:tls-verification-test-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] name: tls-verification-test-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: 27091116-f7ff-44d1-a55c-f16dd2ec9c16 [e2e-llm-inference-service] resourceVersion: '80148' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - patch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - events [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] uid: f4a39702-f18b-44d2-9a3b-63e67841ff6a [e2e-llm-inference-service] resourceVersion: '80102' [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T23:09:58Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80498' [e2e-llm-inference-service] uid: 4c9fc98c-ffef-4442-ad56-0edf977592e7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:47Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T23:09:58Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80498' [e2e-llm-inference-service] uid: 4c9fc98c-ffef-4442-ad56-0edf977592e7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:47Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80474' [e2e-llm-inference-service] uid: a26c7db0-8454-4c96-974c-f502ae5b9eac [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:48Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:48Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-09-18T23:09:50Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80407' [e2e-llm-inference-service] uid: 509c488f-8904-498f-96df-41bb45bceb59 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: tls-verification-test-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:49Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-09-18T23:09:50Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80208' [e2e-llm-inference-service] uid: ef0bd4ab-4a59-47e9-a5fe-65acb29203ad [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80480' [e2e-llm-inference-service] uid: 7e6be3fe-b6f3-480a-b6a6-59303582506b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-inference-pool-ip-e386aca5.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80219' [e2e-llm-inference-service] uid: 31cb2c74-73dd-452c-ab2f-a80d5ab0bd15 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80208' [e2e-llm-inference-service] uid: ef0bd4ab-4a59-47e9-a5fe-65acb29203ad [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80480' [e2e-llm-inference-service] uid: 7e6be3fe-b6f3-480a-b6a6-59303582506b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-inference-pool-ip-e386aca5.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80219' [e2e-llm-inference-service] uid: 31cb2c74-73dd-452c-ab2f-a80d5ab0bd15 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80208' [e2e-llm-inference-service] uid: ef0bd4ab-4a59-47e9-a5fe-65acb29203ad [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-epp-service.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:57Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80480' [e2e-llm-inference-service] uid: 7e6be3fe-b6f3-480a-b6a6-59303582506b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-inference-pool-ip-e386aca5.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:46Z' [e2e-llm-inference-service] name: tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80219' [e2e-llm-inference-service] uid: 31cb2c74-73dd-452c-ab2f-a80d5ab0bd15 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: tls-verification-test-kserve-workload-svc.e2e-test-llm-tls-resources-e4cad0c3.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"89623d06-ad78-4b7b-ac9b-7420cfa1b087"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-09-18T23:09:45Z' [e2e-llm-inference-service] name: tls-verification-test-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: tls-verification-test [e2e-llm-inference-service] uid: 89623d06-ad78-4b7b-ac9b-7420cfa1b087 [e2e-llm-inference-service] resourceVersion: '80167' [e2e-llm-inference-service] uid: 8e427bc9-46e7-4b6e-ad91-e0eb67039702 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: tls-verification-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-56b598698f-4h4fv [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:27:15Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 56b598698f [e2e-llm-inference-service] timestamp: '2026-09-18T23:26:54Z' [e2e-llm-inference-service] window: 17.116s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 14957253n [e2e-llm-inference-service] memory: 26248Ki [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 100157688n [e2e-llm-inference-service] memory: 2406076Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-prefill-7f8db478bb-9l47n [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:27:15Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7f8db478bb [e2e-llm-inference-service] timestamp: '2026-09-18T23:27:11Z' [e2e-llm-inference-service] window: 26.737s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 95533231n [e2e-llm-inference-service] memory: 2405048Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:296 --- [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_tls:diagnostic.py:297 metadata: [e2e-llm-inference-service] name: tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh [e2e-llm-inference-service] namespace: e2e-test-llm-tls-resources-e4cad0c3 [e2e-llm-inference-service] creationTimestamp: '2026-09-18T23:27:15Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: tls-verification-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 78cc9f8576 [e2e-llm-inference-service] timestamp: '2026-09-18T23:27:02Z' [e2e-llm-inference-service] window: 17.689s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 109872123n [e2e-llm-inference-service] memory: 33972Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-09-18T23:27:15.744561] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'tls-verification-test', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-tls-resources-e4cad0c3', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-tls-verification-86160844'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-tls-verificatio-cc747c7a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-tls-verificat-7c5fca28'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: 3 pod(s) for tls-verification-test still terminating: ['tls-verification-test-kserve-56b598698f-4h4fv', 'tls-verification-test-kserve-prefill-7f8db478bb-9l47n', 'tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh'] [e2e-llm-inference-service] assert not ['tls-verification-test-kserve-56b598698f-4h4fv', 'tls-verification-test-kserve-prefill-7f8db478bb-9l47n', 'tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1450 Waiting: 1 pod(s) for tls-verification-test still terminating: ['tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh'] [e2e-llm-inference-service] assert not ['tls-verification-test-kserve-router-scheduler-78cc9f8576-5r8gh'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-09-18T23:28:05.898706] end - ✅ in 50.154s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_tls_resources] [2026-09-18T23:28:05.898872] end - ❌ 1127.192s: assert '--ssl-ciphers' not in '/bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, find the most common IP [e2e-llm-inference-service] v4 RoCE v2 GID index across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there\'s a tie, prefer index 3 as it\'s most common in SR-IOV setups\n if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_index_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-proj [e2e-llm-inference-service] ect/vllm#30937).\n# ODH supports older accelerator images, so only pass it when available.\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Decode is the KV consumer; without a connector here it recomputes prefill\'s KV.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_consumer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \\\n --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \\\n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gi [e2e-llm-inference-service] d_attrs/types/* 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there\'s a tie, prefer ind [e2e-llm-inference-service] ex 3 as it\'s most common in SR-IOV setups\n if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_index_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n# ODH supports older accelerator images, so only pass it when available.\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Prefill is the KV producer; without a connector here decode has nothing to fetch.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_producer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \\\n --served-model-n [e2e-llm-inference-service] ame "facebook/opt-125m" \\\n --port 8000 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" --' [e2e-llm-inference-service] [e2e-llm-inference-service] '--ssl-ciphers' is contained here: [e2e-llm-inference-service] /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] ARCH="$(arch)" [e2e-llm-inference-service] case "${ARCH}" in [e2e-llm-inference-service] ppc64le) [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] s390x) [e2e-llm-inference-service] export FLEX_DEVICE=VF [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] fi [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] x86_64) [e2e-llm-inference-service] export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] ;; [e2e-llm-inference-service] esac [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] [e2e-llm-inference-service] cat /proc/driver/nvidia/params [e2e-llm-inference-service] [e2e-llm-inference-service] KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] active_hcas=() [e2e-llm-inference-service] # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] # Ensure it's a directory before proceeding [e2e-llm-inference-service] if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] active_hcas+=("$hca_name") [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if we found any active HCAs [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] # Join the array elements with a comma [e2e-llm-inference-service] hca_port_pairs=() [e2e-llm-inference-service] for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] [e2e-llm-inference-service] # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] declare -A gid_index_count [e2e-llm-inference-service] declare -A hca_gid_index [e2e-llm-inference-service] [e2e-llm-inference-service] for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] [e2e-llm-inference-service] # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] idx=$(basename "$tpath") [e2e-llm-inference-service] gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] best_gid_index="" [e2e-llm-inference-service] max_count=0 [e2e-llm-inference-service] for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] if [ $count -gt $max_count ]; then [e2e-llm-inference-service] max_count=$count [e2e-llm-inference-service] best_gid_index="$idx" [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] best_gid_index="3" [e2e-llm-inference-service] echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] [e2e-llm-inference-service] export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] fi [e2e-llm-inference-service] echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] [e2e-llm-inference-service] # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] ? +++++++++++++ [e2e-llm-inference-service] # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] fi [e2e-llm-inference-service] # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] # Decode is the KV consumer; without a connector here it recomputes prefill's KV. [e2e-llm-inference-service] if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\"}'" [e2e-llm-inference-service] echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \ [e2e-llm-inference-service] --port 8001 \ [e2e-llm-inference-service] --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] --enable-ssl-refresh \ [e2e-llm-inference-service] --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] ARCH="$(arch)" [e2e-llm-inference-service] case "${ARCH}" in [e2e-llm-inference-service] ppc64le) [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] s390x) [e2e-llm-inference-service] export FLEX_DEVICE=VF [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] fi [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] x86_64) [e2e-llm-inference-service] export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] ;; [e2e-llm-inference-service] esac [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] [e2e-llm-inference-service] cat /proc/driver/nvidia/params [e2e-llm-inference-service] [e2e-llm-inference-service] KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] active_hcas=() [e2e-llm-inference-service] # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] # Ensure it's a directory before proceeding [e2e-llm-inference-service] if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] active_hcas+=("$hca_name") [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if we found any active HCAs [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] # Join the array elements with a comma [e2e-llm-inference-service] hca_port_pairs=() [e2e-llm-inference-service] for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] [e2e-llm-inference-service] # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] declare -A gid_index_count [e2e-llm-inference-service] declare -A hca_gid_index [e2e-llm-inference-service] [e2e-llm-inference-service] for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] [e2e-llm-inference-service] # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] idx=$(basename "$tpath") [e2e-llm-inference-service] gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] best_gid_index="" [e2e-llm-inference-service] max_count=0 [e2e-llm-inference-service] for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] if [ $count -gt $max_count ]; then [e2e-llm-inference-service] max_count=$count [e2e-llm-inference-service] best_gid_index="$idx" [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] best_gid_index="3" [e2e-llm-inference-service] echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] [e2e-llm-inference-service] export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] fi [e2e-llm-inference-service] echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] [e2e-llm-inference-service] # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] fi [e2e-llm-inference-service] # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] # Prefill is the KV producer; without a connector here decode has nothing to fetch. [e2e-llm-inference-service] if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\"}'" [e2e-llm-inference-service] echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] --served-model-name "facebook/opt-125m" \ [e2e-llm-inference-service] --port 8000 \ [e2e-llm-inference-service] --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] --enable-ssl-refresh \ [e2e-llm-inference-service] --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] $@" -- [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:190 Skipping deletion of namespace e2e-test-llm-tls-resources-e4cad0c3 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:977: 246 warnings [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:977: PytestDeprecationWarning: The "scope" keyword argument to the asyncio marker has been deprecated. Please use the "loop_scope" argument instead. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn(PytestDeprecationWarning(_MARKER_SCOPE_KWARG_DEPRECATION_WARNING)) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-oci] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-template-args-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:267: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:153: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] - generated xml file: /workspace/artifacts-dir/junit_e2e-llmisvc_core_and_cluster_cpu_and_not_pvc_storage.xml - [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results-llmisvc_core_and_cluster_cpu_and_not_pvc_storage.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] - AssertionError: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'RouterReady', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-09-18T22:07:42Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:25Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:17:24Z', 'message': 'ReplicaSet "fc-concurrency-test-kserve-router-scheduler-c6579d6db" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:07:39Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] - AssertionError: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:08:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:18:34Z', 'message': 'ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-6db4b76ccf" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:08:46Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] - AssertionError: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-09-18T22:24:23Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:23:53Z', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-09-18T22:33:52Z', 'message': 'ReplicaSet "scheduler-v06-threshold-migration-test-kserve-router-scheduler-598d8f8cc9" has timed out progressing.', 'reason': 'ProgressDeadlineExceeded', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-09-18T22:24:19Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] FAILED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] - assert '--ssl-ciphers' not in '/bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, fi [e2e-llm-inference-service] nd the most common IPv4 RoCE v2 GID index across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there\'s a tie, prefer index 3 as it\'s most common in SR-IOV setups\n if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_index_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in v [e2e-llm-inference-service] LLM 0.15.0 (vllm-project/vllm#30937).\n# ODH supports older accelerator images, so only pass it when available.\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Decode is the KV consumer; without a connector here it recomputes prefill\'s KV.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_consumer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \\\n --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \\\n --port 8001 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x\nif [ -d /opt/ibm/spyre ]; then\n ARCH="$(arch)"\n case "${ARCH}" in\n ppc64le)\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n s390x)\n export FLEX_DEVICE=VF\n if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n fi\n export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/\n if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then\n if jq -e \'(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)\' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then\n export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json"\n else\n echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation"\n rm -f "$HOME/.senlib.json"\n fi\n else\n echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation"\n fi\n if [ -f /opt/rh/gcc-toolset-14/enable ]; then\n . /opt/rh/gcc-toolset-14/enable\n export PATH\n fi\n ;;\n x86_64)\n export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0\n ;;\n esac\nfi\n\nif [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\nfi\n\nif [ "$KSERVE_INFER_ROCE" = "true" ]; then\n echo "Trying to infer RoCE configs ... "\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n grep -H . /sys/class/in [e2e-llm-inference-service] finiband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"}\n\n echo "[Infer RoCE] Discovering active HCAs ..."\n active_hcas=()\n # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*; do\n # Ensure it\'s a directory before proceeding\n if [ -d "$hca_dir" ]; then\n hca_name=$(basename "$hca_dir")\n port_state_file="$hca_dir/ports/1/state" # Assume port 1\n type_file="$hca_dir/ports/1/gid_attrs/types/*"\n\n echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains \'ACTIVE\'"\n if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then\n echo "[Infer RoCE] Found active HCA: $hca_name"\n active_hcas+=("$hca_name")\n else\n echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name"\n fi\n fi\n done\n\n # Check if we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with a comma\n hca_port_pairs=()\n for hca in "${active_hcas[@]}"; do\n hca_port_pairs+=("${hca}:1")\n done\n\n active_hca_list=$(IFS=,; echo "${active_hcas[*]}")\n hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}")\n echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}"\n export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}"\n echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n else\n echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set."\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..."\n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name in "${active_hcas[@]}"; do\n echo "[Infer RoCE] Processing HCA: ${hca_name}"\n\n # Find all RoCE v2 IPv4 GIDs for this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do\n if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then\n idx=$(basename "$tpath")\n gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n # Check for IPv4 GID (contains ffff:)\n if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then\n gid_value=$(cat "$gid_file" 2>/dev/null || echo "")\n echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}"\n hca_gid_index["${hca_name}"]="${idx}"\n gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1))\n break # Use first found IPv4 GID per HCA\n fi\n fi\n done\n done\n\n # Find the most common GID index (most likely to be consistent across nodes)\n best_gid_index=""\n max_count=0\n for idx in "${!gid_index_count[@]}"; do\n count=${gid_index_count["${idx}"]}\n echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs"\n if [ $count -gt $max_count ]; then\n max_count=$count\n best_gid_index="$idx"\n fi\n done\n\n # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}"\n # If there [e2e-llm-inference-service] \'s a tie, prefer index 3 as it\'s most common in SR-IOV setups\n if [ -n "${gid_index_count[\'3\']}" ] && [ "${gid_index_count[\'3\']}" -eq "$max_count" ]; then\n best_gid_index="3"\n echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)"\n fi\n fi\n\n # Check if GID_INDEX is already set via environment variables\n if [ -n "${NCCL_IB_GID_INDEX}" ]; then\n echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX"\n elif [ -n "$best_gid_index" ]; then\n echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX"\n else\n echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA."\n fi\n else\n echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference."\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS="--disable-uvicorn-access-log"\nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk \'{print $NF}\')\necho "[access-log-detect] vllm version=\'${VLLM_VERSION}\'"\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then\n ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping"\nfi\necho "[access-log-detect] selected ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n\n# --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937).\n# ODH supports older accelerator images, so only pass it when available.\n\n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\nSHUTDOWN_TIMEOUT_ARGS=""\nif [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then\n SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40"\nfi\n\n# A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset.\nKV_TRANSFER_ARGS=""\nif [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then\n # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020).\n if [[ "$VLLM_VERSION" =~ ^[0-9]+\\.[0-9]+ ]] && [ "$(printf \'%s\\n%s\\n\' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then\n KV_TRANSFER_ARGS=""\n fi\n # This template is only composed for a disaggregated P/D topology (spec.prefill set).\n # Prefill is the KV producer; without a connector here decode has nothing to fetch.\n if [ -z "${KV_TRANSFER_ARGS}" ]; then\n # Only inject when NIXL is importable. CPU and other non-NIXL engine images\n # raise RuntimeError("NIXL is not available") and never finish starting.\n NIXL_PY=$(command -v python3 || command -v python || true)\n if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then\n KV_TRANSFER_ARGS="--kv-transfer-config \'{\\"kv_connector\\":\\"NixlConnector\\",\\"kv_role\\":\\"kv_producer\\"}\'"\n echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)"\n else\n echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled"\n fi\n fi\nfi\n\neval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] \\n --served-model-name "facebook/opt-125m" \\\n --port 8000 \\\n --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@" --' [e2e-llm-inference-service] [e2e-llm-inference-service] '--ssl-ciphers' is contained here: [e2e-llm-inference-service] /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] ARCH="$(arch)" [e2e-llm-inference-service] case "${ARCH}" in [e2e-llm-inference-service] ppc64le) [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] s390x) [e2e-llm-inference-service] export FLEX_DEVICE=VF [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] fi [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] x86_64) [e2e-llm-inference-service] export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] ;; [e2e-llm-inference-service] esac [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] [e2e-llm-inference-service] cat /proc/driver/nvidia/params [e2e-llm-inference-service] [e2e-llm-inference-service] KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] active_hcas=() [e2e-llm-inference-service] # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] # Ensure it's a directory before proceeding [e2e-llm-inference-service] if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] active_hcas+=("$hca_name") [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if we found any active HCAs [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] # Join the array elements with a comma [e2e-llm-inference-service] hca_port_pairs=() [e2e-llm-inference-service] for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] [e2e-llm-inference-service] # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] declare -A gid_index_count [e2e-llm-inference-service] declare -A hca_gid_index [e2e-llm-inference-service] [e2e-llm-inference-service] for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] [e2e-llm-inference-service] # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] idx=$(basename "$tpath") [e2e-llm-inference-service] gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] best_gid_index="" [e2e-llm-inference-service] max_count=0 [e2e-llm-inference-service] for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] if [ $count -gt $max_count ]; then [e2e-llm-inference-service] max_count=$count [e2e-llm-inference-service] best_gid_index="$idx" [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] best_gid_index="3" [e2e-llm-inference-service] echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] [e2e-llm-inference-service] export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] fi [e2e-llm-inference-service] echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] [e2e-llm-inference-service] # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] ? +++++++++++++ [e2e-llm-inference-service] # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] fi [e2e-llm-inference-service] # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] # Decode is the KV consumer; without a connector here it recomputes prefill's KV. [e2e-llm-inference-service] if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\"}'" [e2e-llm-inference-service] echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_consumer)" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] --served-model-name "facebook/opt-125m" "publishers/e2e-test-llm-tls-resources-e4cad0c3/models/facebook/opt-125m" \ [e2e-llm-inference-service] --port 8001 \ [e2e-llm-inference-service] --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] --enable-ssl-refresh \ [e2e-llm-inference-service] --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] $@" -- /bin/bash -c # Spyre architecture-specific setup for ppc64le/s390x [e2e-llm-inference-service] if [ -d /opt/ibm/spyre ]; then [e2e-llm-inference-service] ARCH="$(arch)" [e2e-llm-inference-service] case "${ARCH}" in [e2e-llm-inference-service] ppc64le) [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] s390x) [e2e-llm-inference-service] export FLEX_DEVICE=VF [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] export TORCH_SENDNN_TEMP_CACHE_DIR=/opt/ibm/spyre/models/cache/ [e2e-llm-inference-service] if [ -n "${AIU_AUTOGEN_SENLIB_CONFIG_FILE:-}" ] && [ -r "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" ]; then [e2e-llm-inference-service] if jq -e '(.SNT_MCI.DCR.MCI_CTRL.ENABLE_RISCV = "0x0") | del(.SNT_MCI.init) | (.METRICS.general.enable = true)' "${AIU_AUTOGEN_SENLIB_CONFIG_FILE}" > "$HOME/.senlib.json" && [ -s "$HOME/.senlib.json" ]; then [e2e-llm-inference-service] export SENLIB_DEVEL_CONFIG_FILE="$HOME/.senlib.json" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: jq failed to process ${AIU_AUTOGEN_SENLIB_CONFIG_FILE}, skipping SENLIB config generation" [e2e-llm-inference-service] rm -f "$HOME/.senlib.json" [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "WARNING: AIU_AUTOGEN_SENLIB_CONFIG_FILE is not set or not readable, skipping SENLIB config generation" [e2e-llm-inference-service] fi [e2e-llm-inference-service] if [ -f /opt/rh/gcc-toolset-14/enable ]; then [e2e-llm-inference-service] . /opt/rh/gcc-toolset-14/enable [e2e-llm-inference-service] export PATH [e2e-llm-inference-service] fi [e2e-llm-inference-service] ;; [e2e-llm-inference-service] x86_64) [e2e-llm-inference-service] export SENDNN_INFERENCE_REQUIRE_PRECOMPILED_DECODERS=0 [e2e-llm-inference-service] ;; [e2e-llm-inference-service] esac [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then [e2e-llm-inference-service] source /etc/profile.d/ibm-aiu-setup.sh [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ "$KSERVE_INFER_ROCE" = "true" ]; then [e2e-llm-inference-service] echo "Trying to infer RoCE configs ... " [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null [e2e-llm-inference-service] grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null [e2e-llm-inference-service] [e2e-llm-inference-service] cat /proc/driver/nvidia/params [e2e-llm-inference-service] [e2e-llm-inference-service] KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE v2"} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Discovering active HCAs ..." [e2e-llm-inference-service] active_hcas=() [e2e-llm-inference-service] # Loop through all mlx5 devices found in sysfs [e2e-llm-inference-service] for hca_dir in /sys/class/infiniband/mlx5_*; do [e2e-llm-inference-service] # Ensure it's a directory before proceeding [e2e-llm-inference-service] if [ -d "$hca_dir" ]; then [e2e-llm-inference-service] hca_name=$(basename "$hca_dir") [e2e-llm-inference-service] port_state_file="$hca_dir/ports/1/state" # Assume port 1 [e2e-llm-inference-service] type_file="$hca_dir/ports/1/gid_attrs/types/*" [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Check if the port state file ${port_state_file} exists and contains 'ACTIVE'" [e2e-llm-inference-service] if [ -f "$port_state_file" ] && grep -q "ACTIVE" "$port_state_file" && grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" ${type_file} 2>/dev/null; then [e2e-llm-inference-service] echo "[Infer RoCE] Found active HCA: $hca_name" [e2e-llm-inference-service] active_hcas+=("$hca_name") [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] Skipping inactive or down HCA: $hca_name" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if we found any active HCAs [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] # Join the array elements with a comma [e2e-llm-inference-service] hca_port_pairs=() [e2e-llm-inference-service] for hca in "${active_hcas[@]}"; do [e2e-llm-inference-service] hca_port_pairs+=("${hca}:1") [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] active_hca_list=$(IFS=,; echo "${active_hcas[*]}") [e2e-llm-inference-service] hca_port_pairs_list=$(IFS=,; echo "${hca_port_pairs[*]}") [e2e-llm-inference-service] echo "[Infer RoCE] Setting active HCAs: ${active_hca_list}" [e2e-llm-inference-service] export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}} [e2e-llm-inference-service] export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}} [e2e-llm-inference-service] export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}" [e2e-llm-inference-service] echo "[Infer RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}" [e2e-llm-inference-service] echo "[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA will not be set." [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] if [ ${#active_hcas[@]} -gt 0 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)..." [e2e-llm-inference-service] [e2e-llm-inference-service] # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index across all HCAs [e2e-llm-inference-service] declare -A gid_index_count [e2e-llm-inference-service] declare -A hca_gid_index [e2e-llm-inference-service] [e2e-llm-inference-service] for hca_name in "${active_hcas[@]}"; do [e2e-llm-inference-service] echo "[Infer RoCE] Processing HCA: ${hca_name}" [e2e-llm-inference-service] [e2e-llm-inference-service] # Find all RoCE v2 IPv4 GIDs for this HCA and count by index [e2e-llm-inference-service] for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; do [e2e-llm-inference-service] if grep -q "${KSERVE_INFER_IB_GID_INDEX_GREP}" "$tpath" 2>/dev/null; then [e2e-llm-inference-service] idx=$(basename "$tpath") [e2e-llm-inference-service] gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}" [e2e-llm-inference-service] # Check for IPv4 GID (contains ffff:) [e2e-llm-inference-service] if [ -f "$gid_file" ] && grep -q "ffff:" "$gid_file"; then [e2e-llm-inference-service] gid_value=$(cat "$gid_file" 2>/dev/null || echo "") [e2e-llm-inference-service] echo "[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}: index=${idx}, gid=${gid_value}" [e2e-llm-inference-service] hca_gid_index["${hca_name}"]="${idx}" [e2e-llm-inference-service] gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} + 1)) [e2e-llm-inference-service] break # Use first found IPv4 GID per HCA [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Find the most common GID index (most likely to be consistent across nodes) [e2e-llm-inference-service] best_gid_index="" [e2e-llm-inference-service] max_count=0 [e2e-llm-inference-service] for idx in "${!gid_index_count[@]}"; do [e2e-llm-inference-service] count=${gid_index_count["${idx}"]} [e2e-llm-inference-service] echo "[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs" [e2e-llm-inference-service] if [ $count -gt $max_count ]; then [e2e-llm-inference-service] max_count=$count [e2e-llm-inference-service] best_gid_index="$idx" [e2e-llm-inference-service] fi [e2e-llm-inference-service] done [e2e-llm-inference-service] [e2e-llm-inference-service] # Use deterministic fallback if tied - prefer index 3 (SR-IOV standard) [e2e-llm-inference-service] if [ ${#gid_index_count[@]} -gt 1 ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Multiple GID indices found, selecting most common: ${best_gid_index}" [e2e-llm-inference-service] # If there's a tie, prefer index 3 as it's most common in SR-IOV setups [e2e-llm-inference-service] if [ -n "${gid_index_count['3']}" ] && [ "${gid_index_count['3']}" -eq "$max_count" ]; then [e2e-llm-inference-service] best_gid_index="3" [e2e-llm-inference-service] echo "[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV standard)" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # Check if GID_INDEX is already set via environment variables [e2e-llm-inference-service] if [ -n "${NCCL_IB_GID_INDEX}" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} from environment" [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX} [e2e-llm-inference-service] echo "[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] elif [ -n "$best_gid_index" ]; then [e2e-llm-inference-service] echo "[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found on ${max_count} HCAs)" [e2e-llm-inference-service] [e2e-llm-inference-service] export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index} [e2e-llm-inference-service] [e2e-llm-inference-service] echo "[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL, NVSHMEM, and UCX" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] ERROR: No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA." [e2e-llm-inference-service] fi [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[Infer RoCE] No active HCAs found, skipping GID_INDEX inference." [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # --disable-access-log-for-endpoints landed in vLLM 0.16.0 (vllm-project/vllm#30011). [e2e-llm-inference-service] # Older versions still need the blanket --disable-uvicorn-access-log. [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-uvicorn-access-log" [e2e-llm-inference-service] VLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}') [e2e-llm-inference-service] echo "[access-log-detect] vllm version='${VLLM_VERSION}'" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.16.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.16.0" ]; then [e2e-llm-inference-service] ACCESS_LOG_ARGS="--disable-access-log-for-endpoints /health,/metrics,/ping" [e2e-llm-inference-service] fi [e2e-llm-inference-service] echo "[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'" [e2e-llm-inference-service] [e2e-llm-inference-service] # --ssl-ciphers landed in vLLM 0.15.0 (vllm-project/vllm#30937). [e2e-llm-inference-service] # ODH supports older accelerator images, so only pass it when available. [e2e-llm-inference-service] [e2e-llm-inference-service] [e2e-llm-inference-service] # --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666). [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="" [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.18.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.18.0" ]; then [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout 40" [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] # A user-supplied --kv-transfer-config always wins; KServe only fills the flag when it is unset. [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] if [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv-transfer-config"* ]] && [[ "${VLLM_ADDITIONAL_ARGS:-}" != *"--kv_transfer_config"* ]] && [[ "$*" != *"--kv-transfer-config"* ]] && [[ "$*" != *"--kv_transfer_config"* ]]; then [e2e-llm-inference-service] # --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+ (vllm-project/vllm#40020). [e2e-llm-inference-service] if [[ "$VLLM_VERSION" =~ ^[0-9]+\.[0-9]+ ]] && [ "$(printf '%s\n%s\n' "0.22.0" "${VLLM_VERSION}" | sort -V | head -1)" = "0.22.0" ]; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="" [e2e-llm-inference-service] fi [e2e-llm-inference-service] # This template is only composed for a disaggregated P/D topology (spec.prefill set). [e2e-llm-inference-service] # Prefill is the KV producer; without a connector here decode has nothing to fetch. [e2e-llm-inference-service] if [ -z "${KV_TRANSFER_ARGS}" ]; then [e2e-llm-inference-service] # Only inject when NIXL is importable. CPU and other non-NIXL engine images [e2e-llm-inference-service] # raise RuntimeError("NIXL is not available") and never finish starting. [e2e-llm-inference-service] NIXL_PY=$(command -v python3 || command -v python || true) [e2e-llm-inference-service] if [ -n "${NIXL_PY}" ] && "${NIXL_PY}" -c "import nixl" >/dev/null 2>&1; then [e2e-llm-inference-service] KV_TRANSFER_ARGS="--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\"}'" [e2e-llm-inference-service] echo "[kv-transfer] NIXL available, enabling NixlConnector (kv_producer)" [e2e-llm-inference-service] else [e2e-llm-inference-service] echo "[kv-transfer] NIXL not available, P/D KV transfer stays disabled" [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] fi [e2e-llm-inference-service] [e2e-llm-inference-service] eval "exec vllm serve /mnt/models \ [e2e-llm-inference-service] --served-model-name "facebook/opt-125m" \ [e2e-llm-inference-service] --port 8000 \ [e2e-llm-inference-service] --root-path /e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test \ [e2e-llm-inference-service] ${ACCESS_LOG_ARGS} \ [e2e-llm-inference-service] ${SHUTDOWN_TIMEOUT_ARGS} \ [e2e-llm-inference-service] ${KV_TRANSFER_ARGS} \ [e2e-llm-inference-service] --enable-ssl-refresh \ [e2e-llm-inference-service] --ssl-certfile /var/run/kserve/tls/tls.crt \ [e2e-llm-inference-service] --ssl-keyfile /var/run/kserve/tls/tls.key \ [e2e-llm-inference-service] ${VLLM_ADDITIONAL_ARGS} \ [e2e-llm-inference-service] $@" -- [e2e-llm-inference-service] ====== 4 failed, 36 passed, 3 skipped, 275 warnings in 5297.34s (1:28:17) ====== [e2e-llm-inference-service] make[1]: *** [Makefile.ocp.mk:85: run-e2e-ocp] Error 1 [e2e-llm-inference-service] make[1]: Leaving directory '/workspace/source' [e2e-llm-inference-service] make: *** [Makefile.ocp.mk:106: e2e-llmisvc-ocp] Error 2 [must-gather] [must-gather ] OUT 2026-09-18T23:33:10.892011307Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 6efe82c2-d17a-4b5f-be53-03909b4cd2d6 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.34" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-09-18T23:33:10.928056137Z namespace/openshift-must-gather-wgt88 created [must-gather] [must-gather ] OUT 2026-09-18T23:33:10.93797808Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-9x2k2 created [must-gather] [must-gather ] OUT 2026-09-18T23:33:10.970457802Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:17.723220639Z [disk usage checker] Started [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:17.726433926Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:17.988858334Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.151262010Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.151411504Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.151411504Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.153676668Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.314831758Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.314861366Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.314861366Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.317000381Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.478495033Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.478525285Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.478525285Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.480839448Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.640736253Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.640767850Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.640767850Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.643274629Z Error getting logs from rhods-notebooks [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.798808269Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.798901787Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.798901787Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.801130729Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.955774557Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.955805950Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.955805950Z namespaces "istio-system" not found [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:18.957829002Z Error getting logs from istio-system [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:19.115631775Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:19.115663903Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:19.115663903Z namespaces "knative-serving" not found [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:19.117876813Z Error getting logs from knative-serving [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:19.277570588Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:19.277665341Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:19.277665341Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:19.279935472Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:19.555150105Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:19.842136959Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:19.987498633Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.068749448Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.070803299Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.201832895Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.288744610Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.291044564Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.421900068Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.501809904Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.504009960Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.639821691Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.720985065Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.722860725Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.858436881Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.941560916Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:20.943554626Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.077575116Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.165253288Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.167544084Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.302899692Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.384383255Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.386776932Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.519169354Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.599937730Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.602059160Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.736044313Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.818043336Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.820043725Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:21.952539501Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.035063907Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.037240586Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.173041785Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.253747497Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.255856945Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.384492865Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.465359673Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.467689860Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.596038357Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.682868092Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.685018291Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.730975144Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.814952371Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.897298100Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:22.899456919Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.030091648Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.113964255Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.116434068Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.251838994Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.333169313Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.335084439Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.470140435Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.551935625Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.554056597Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.683927235Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.768379897Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:23.770476346Z Error collecting info from [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:24.493525939Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:24.623525645Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:24.755434112Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:24.883435398Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:25.011115204Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:25.607433619Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:25.734054810Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:25.868904211Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:26.002974532Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:26.136409843Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:26.685102572Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:26.828659572Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:26.918230172Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:26.988516337Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:27.152788594Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:27.287095958Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:27.385213591Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:27.465661000Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:27.624628528Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:27.735380235Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:27.755458355Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:27.833865600Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:27.908335220Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:28.068943158Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:28.202558276Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:28.281859232Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:28.354011432Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:28.514317177Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:28.651565188Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:28.734280902Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:28.801975396Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:28.965136775Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:29.109931139Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:29.194608271Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:29.268042636Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:29.423564829Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:29.564072587Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:29.640670110Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:29.710430685Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:29.865817838Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:30.001785578Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:30.083454163Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:30.157927728Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:30.317573177Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:30.449085018Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:30.533130327Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:30.603842970Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:30.769242051Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:30.955597808Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:31.039006330Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:31.109720848Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:31.272058673Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:31.424520525Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:31.517060782Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:31.591131566Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:31.753505776Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:31.894072203Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:31.977974188Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:32.065437021Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:32.229063707Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:32.374320775Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:32.457193440Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:32.543556275Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:32.702833191Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:32.740872924Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:32.875535129Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:32.988471971Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:33.066831700Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:33.225292628Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:33.363467162Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:33.446638995Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:33.519027264Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:33.675881168Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:34.026234065Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:34.193835654Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:34.299104416Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:34.457592802Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:34.768145818Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:34.932922331Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:35.041525952Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:35.201082748Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:35.505808696Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:35.653189067Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:35.756221762Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:35.924620312Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:36.301354727Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:36.477013361Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:36.580764648Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:36.739235145Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:37.305016268Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:37.488492685Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:37.611061675Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:37.745517117Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:37.771536936Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:38.557938424Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:38.877153568Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:39.176848967Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:39.339470690Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:40.072100816Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:40.291673980Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:40.422224933Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:40.588964586Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:41.334964417Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:41.518197248Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:41.633842765Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:41.791863099Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:42.390187051Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:42.572962297Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:42.702351322Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:42.750602304Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:42.865769709Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:43.478175171Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:43.679990736Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:43.798920561Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:43.959048520Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:44.535919829Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:44.717064847Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:44.848632642Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:45.020619131Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:45.595088347Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:45.774265253Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:45.895735242Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:46.059460692Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:46.623925193Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:46.813015126Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:46.930159027Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:47.087883807Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:47.757696750Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:47.842565094Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:48.022079177Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:48.145421712Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:48.318554748Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:48.892860003Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:49.081438610Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:49.213135413Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:49.374858331Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:49.939336442Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:50.120145854Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:50.244660853Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:50.421921242Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:51.116162535Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:51.306614828Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:51.432021656Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:51.593950510Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:52.163207180Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:52.350782184Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:52.467897855Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:52.622875136Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:52.763832706Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:53.218117883Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:53.466528732Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:53.590727748Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:53.751483783Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:54.340860893Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:54.518245493Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:54.635515207Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:54.795089327Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:55.368479866Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:55.546723661Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:55.663577969Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:55.828570228Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:56.412087480Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:56.588388657Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:56.709014534Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:56.869431410Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:57.441714342Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:57.622225332Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:57.740676713Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:57.768340950Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:57.907419411Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:58.507309477Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:58.693169691Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:58.810736779Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:58.970817141Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:59.636051284Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:59.829314287Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:33:59.948321587Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:00.106861634Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:00.680486921Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:00.863855452Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:00.986445436Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:01.159416295Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:01.742514976Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:01.932411616Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:02.058625900Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:02.220882810Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:02.772724457Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:02.824675660Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:03.013248758Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:03.129184519Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:03.289819559Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:03.870739890Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:04.049904221Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:04.175229985Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:04.338034706Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:04.924151160Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:05.113835922Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:05.271380119Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:05.429125781Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:06.005910993Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:06.184199566Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:06.305686495Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:06.472267006Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:07.035244865Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:07.215293817Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:07.337739884Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:07.497165112Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:07.777150424Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:08.190272747Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:08.417183221Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:08.535977763Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:08.699046521Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:09.291783161Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:09.489738735Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:09.617381494Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:09.780585039Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:10.361870950Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:10.541263527Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:10.687613893Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:10.854700163Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:11.431758853Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:11.626072787Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:11.750593827Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:11.910038303Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:12.484721724Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:12.662770203Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:12.780640635Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:12.781539408Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:12.937088124Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:13.511874139Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:13.680035904Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:13.795994735Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:13.955245718Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:14.537601385Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:14.718887812Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:14.838372958Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:14.994290903Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:15.546162601Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:15.720770140Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:15.844316125Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:16.002960937Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:16.569460349Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:16.745733767Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:16.865560142Z Wrote inspect data to must-gather. [must-gather] [must-gather-79nv5] POD 2026-09-18T23:34:16.888792891Z Caches written to disk [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:17.895679225Z waiting for gather to complete [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:17.899024257Z downloading gather output [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.239393914Z receiving incremental file list [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.251286411Z ./ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.251424495Z aggregated-discovery-api.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.251807845Z aggregated-discovery-apis.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.252879635Z event-filter.html [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.253292146Z timestamp [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.25344702Z version [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.25999074Z cluster-scoped-resources/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260006701Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260012791Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260056702Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260140974Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260151915Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260184086Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260239387Z namespaces/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260248147Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260291089Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260369691Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260390131Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260451613Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260469823Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260572766Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260647008Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.26072901Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260773612Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260802483Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260883555Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.260916816Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.261011708Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.26105115Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.26108746Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.261178993Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.261254565Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.261287606Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.261412049Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.261488882Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.261569424Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.261649996Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.261747269Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.261924914Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262138219Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262184641Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262230962Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262260313Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262320824Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262372556Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262417417Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262495299Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262542411Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262573031Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262630393Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262657674Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262715855Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262775327Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262823848Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.2628841Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262940591Z namespaces/kuadrant-system/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.262987873Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.263043034Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.263107866Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.263173288Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.263228629Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.263334312Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.263670231Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.264014071Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.264069362Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.264106714Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.264143744Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.264192416Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.264277618Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.264313269Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.26435881Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.264439252Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.264485564Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.264512665Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.26470483Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.264856744Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.26543177Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.265527113Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.265926444Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.266062877Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.266299274Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.266444338Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.266502529Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.266649593Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.266683044Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.266727235Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.266772556Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.266858009Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.266927141Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.266944301Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.267002923Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.267047634Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.267095176Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.267105656Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.267118556Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino-6d9898bd99-7df2d.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.267212759Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.267235169Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/authorino/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.26724046Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/authorino/logs/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.26724863Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/authorino/logs/current.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281192923Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281245305Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/authorino/logs/previous.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281268145Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281346417Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/authorino-operator-6fc8f864f4-4srxn.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.28144648Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.28146342Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/manager/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281477021Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/manager/logs/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281492281Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/manager/logs/current.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281662726Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281716987Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/manager/logs/previous.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281770829Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.28179158Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/dns-operator-controller-manager-65dc8f76f5-4sgbx.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281934993Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281955214Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/manager/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281960824Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/manager/logs/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.281971925Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/manager/logs/current.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282094578Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.28217595Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/manager/logs/previous.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282208521Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282272213Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin-fbdd4f445-csmlv.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282342515Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282351585Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282356935Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282400966Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282510499Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282618542Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282630703Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282689194Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/kuadrant-operator-controller-manager-6d4cb79c-vfzqt.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282806877Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282815628Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/manager/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282820458Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/manager/logs/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.282852919Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/manager/logs/current.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.300818583Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.300925366Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/manager/logs/previous.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.300950156Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301006728Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador-limitador-84994f7ffc-2z8v7.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301103891Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301114391Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/limitador/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301119341Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/limitador/logs/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301164292Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/limitador/logs/current.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301270545Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301333287Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/limitador/logs/previous.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301365908Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301412999Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301501691Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301511342Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/manager/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301516292Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/manager/logs/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301538723Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/manager/logs/current.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301714487Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301787849Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/manager/logs/previous.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.30180408Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301864061Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301923173Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.301982705Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.302024766Z namespaces/openshift-ingress/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.302095848Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.302135639Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.302194201Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.302230882Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.302280993Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.302359795Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.302593212Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.30289613Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.302957431Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.303004613Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.303081665Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.303124606Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.303208258Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.303252679Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.303293131Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.303373073Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.303420684Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.303464395Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.303889307Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.30401059Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.304263367Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.304340809Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.304671638Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.304824633Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.30507981Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305189343Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305243924Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305334147Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305378108Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305421899Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.30546555Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305546913Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305590734Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305635285Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305687086Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305731708Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305774639Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305782609Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.30580553Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/istiod-openshift-gateway-5677ccb49b-zhst7.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305950374Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305964104Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305968174Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.305979514Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/current.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.309478881Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.309549363Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/previous.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.309565653Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.309624595Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.309732888Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.309747758Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.309754988Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.309766838Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.310073557Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.310153139Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.31017692Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router-default-657b5f585b-96g68.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.310300643Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.310320604Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.310328414Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.310346585Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/current.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.310441067Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/previous.insecure.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.310520509Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/previous.log [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.31054348Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.31055474Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.310633992Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.310650843Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.316866104Z [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.316888544Z sent 2,836 bytes received 753,437 bytes 504,182.00 bytes/sec [must-gather] [must-gather-79nv5] OUT 2026-09-18T23:34:18.316905835Z total size is 15,754,083 speedup is 20.83 [must-gather] [must-gather ] OUT 2026-09-18T23:34:18.371048843Z namespace/openshift-must-gather-wgt88 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 6efe82c2-d17a-4b5f-be53-03909b4cd2d6 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.34" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-09-18T23:34:18.482534178Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e89d9d3fe5dc08e644c7ac9f331dd64a8026af5bc1d556e74b5403d9a857d0a8 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 6efe82c2-d17a-4b5f-be53-03909b4cd2d6 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.34" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-09-18T23:34:18.500612675Z namespace/openshift-must-gather-s6w2r created [must-gather] [must-gather ] OUT 2026-09-18T23:34:18.508092831Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-24hsv created [must-gather] [must-gather ] OUT 2026-09-18T23:34:18.526492736Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e89d9d3fe5dc08e644c7ac9f331dd64a8026af5bc1d556e74b5403d9a857d0a8 created [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:20.010219896Z [disk usage checker] Started [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:20.013477936Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:20.326467611Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:20.484176453Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:20.589656908Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:20.695806648Z Gathering data for ns/default... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:21.067424051Z Gathering data for ns/openshift... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:21.556243111Z Gathering data for ns/kube-system... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:22.359157085Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:22.703731014Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:22.703763570Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:22.703763570Z namespaces "assisted-installer" not found [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.169765143Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.174896589Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.177599588Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.182566091Z Executing Istio gather script [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.186538863Z INFO: Gathering HAProxy config files [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.202042006Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.221254517Z INFO: Collecting host service logs for crio [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.221658977Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.222029102Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.222391434Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.222740834Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.223081666Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.223431449Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.223770150Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.224125475Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.224460181Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.224790492Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.225267452Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.242290025Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.663026694Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.864799814Z INFO: Collecting Insights Archives from insights-operator-6f9c644dd9-ws6nf [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.864848908Z insights-runtime-extractor-6jk4g [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.864866907Z insights-runtime-extractor-tv5mn [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:23.864884788Z insights-runtime-extractor-xbczd [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:24.188645897Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:24.188730303Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:24.190871830Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:24.260215455Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:24.349131339Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:24.531337607Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:24.583926893Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:24.593631763Z No resources found [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:24.722535257Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:24.808499280Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.000293429Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.018968567Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.157682420Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.408279345Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.457152285Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.527490224Z No resources found [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.574524922Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.585130235Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.726900189Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.758966203Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.819436858Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.819436858Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.932609426Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:25.991776505Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:26.114122749Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:26.150250414Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:26.181701855Z tar: Removing leading `/' from member names [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:26.190360428Z INFO: HAProxy config collection complete. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:26.207752624Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:26.421458447Z INFO: 'previous-logs' folder not found on ip-10-0-130-79.ec2.internal, skipping... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:26.590023505Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:26.638628765Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:26.643752883Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:26.733134740Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:26.854932901Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.096557281Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.112678241Z INFO: 'previous-logs' folder not found on ip-10-0-141-13.ec2.internal, skipping... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.120314779Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.197092796Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.202297103Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.366609543Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.448012910Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.708492095Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.713253289Z INFO: 'previous-logs' folder not found on ip-10-0-142-209.ec2.internal, skipping... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.713300772Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.713335524Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.714016218Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.847211862Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.870171506Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.902774669Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.910868573Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:27.936308263Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.126444480Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.132095256Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.272853585Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.334334430Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.433877187Z INFO: INTERCONNECT MODE [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.433986079Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.499206058Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.516357017Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.521081481Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.598846715Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.758632595Z INFO: Gathering OVN_Northbound from ovnkube-node-9fjdd... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.759003182Z INFO: Gathering OVN_Northbound from ovnkube-node-dvlk7... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.759390958Z INFO: Gathering OVN_Northbound from ovnkube-node-rmj6f... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.759811571Z INFO: Gathering OVN_Southbound from ovnkube-node-9fjdd... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.760138895Z INFO: Gathering OVN_Southbound from ovnkube-node-dvlk7... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.760434446Z INFO: Gathering OVN_Southbound from ovnkube-node-rmj6f... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.788718242Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.853323844Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:28.867795425Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.055744749Z tar: Removing leading `/' from member names [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.156145983Z tar: Removing leading `/' from member names [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.164839881Z tar: Removing leading `/' from member names [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.181039364Z tar: Removing leading `/' from member names [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.182454148Z tar: Removing leading `/' from member names [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.226971359Z tar: Removing leading `/' from member names [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.326105179Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.338494286Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.638408629Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.658961222Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.831559408Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.831697925Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.832207740Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.834336123Z 9.2M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.837475941Z ovnk_database_store/ [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.837536079Z ovnk_database_store/ovnkube-node-dvlk7_sbdb [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.893725901Z ovnk_database_store/ovnkube-node-9fjdd_nbdb [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.914340556Z ovnk_database_store/ovnkube-node-9fjdd_sbdb [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.973021522Z ovnk_database_store/ovnkube-node-dvlk7_nbdb [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.995813494Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:29.997651963Z ovnk_database_store/ovnkube-node-rmj6f_sbdb [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:30.056839828Z ovnk_database_store/ovnkube-node-rmj6f_nbdb [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:30.087188248Z INFO: Network log collection complete. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:30.133894709Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:30.168977740Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:30.374056120Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:30.609879831Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:30.687772453Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:30.690540323Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:31.210169484Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:31.251968573Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0720c8b078cc0cebb01b4c5cde21180aee7c2bb1a0d322b21890c0d3707406ea [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:31.411757084Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:31.470094438Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:31.580634664Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:31.704923493Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:31.759759599Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:31.968883912Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:31.972087840Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:32.237005450Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:32.270313844Z Gathering data for ns/openshift-console... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:32.477725551Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:32.481240410Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:32.682339666Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:32.755078538Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:32.922238888Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:32.933779581Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:32.937264120Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:33.133749891Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:33.363114779Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:33.367929989Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:33.586077142Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:33.586450195Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:33.824376324Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:33.829755492Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:33.924929591Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:33.978274689Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:34.040479756Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:34.238371211Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:34.243620826Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:34.452542660Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:34.643711769Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:34.695501501Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:34.699333081Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:34.935392446Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:35.076686532Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:35.180203535Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:35.190482216Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:35.193750408Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:35.231698605Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:35.390075452Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:35.626211229Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:35.630244568Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:35.713440469Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:35.824123250Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:36.041356532Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:36.044615927Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:36.225858757Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:36.255168109Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:36.403672894Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:36.474042967Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:36.479840294Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:36.677329752Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:36.885629900Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:36.920821801Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:36.924186778Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:37.094378297Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:37.322087137Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:37.325406410Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:37.376696383Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:37.525743046Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:37.755814461Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:37.759136584Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:37.930245391Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:38.129809182Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:38.133758921Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:38.322255141Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:38.521886909Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:38.529914291Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:38.533878033Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:38.712328703Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:38.920752406Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:38.924076278Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:39.154034117Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:39.368015878Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:39.371490330Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:39.563031159Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:39.666386870Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:39.789959824Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:39.794328691Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:39.978717647Z Collecting performance related data for node ip-10-0-130-79.ec2.internal [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:39.987602940Z Collecting performance related data for node ip-10-0-141-13.ec2.internal [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:40.001246211Z Collecting performance related data for node ip-10-0-142-209.ec2.internal [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:40.003582698Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:40.189254079Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:40.280621929Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:40.284750585Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:40.326202811Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:40.522405800Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:40.823006638Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:41.003100428Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:41.156512199Z Gathering data for ns/kserve... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:41.309100396Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:41.775752127Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:42.025855975Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:42.187314961Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:42.331599386Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:42.558584909Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:42.967797856Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:43.212376276Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:43.395537838Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:43.928055691Z Done executing Istio gather script [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:44.302075188Z Gathering data for ns/openshift-config... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:44.635692512Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:45.197165157Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:45.226698578Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:45.583818909Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:45.907326052Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:46.259410868Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:46.601372196Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:46.903241994Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:47.314540251Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:47.778324748Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:48.185553719Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:48.891688244Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:49.013412823Z Collecting kubelet logs for node ip-10-0-130-79.ec2.internal [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:49.015113283Z Collecting kubelet logs for node ip-10-0-141-13.ec2.internal [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:49.016846545Z Collecting kubelet logs for node ip-10-0-142-209.ec2.internal [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:49.442462067Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-s6w2r namespace [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:49.448055606Z INFO: Node performance data collection complete. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:50.203088270Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:50.370191588Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:51.713275496Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:52.050257459Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:52.599961550Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:53.066787779Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:53.557105218Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:53.846850879Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:54.374485698Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:54.735846285Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:55.179782246Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:55.208730994Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70 [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:55.479929825Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:55.804371303Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:56.479153810Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:56.822033683Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:57.182972789Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:57.590378200Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:58.539357120Z Wrote inspect data to must-gather. [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:58.539402193Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:58.539402193Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:58.676781728Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-f7kkf] POD 2026-09-18T23:34:58.858805485Z Caches written to disk [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:34:59.866224899Z waiting for gather to complete [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:34:59.869328515Z downloading gather output [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.131663857Z receiving incremental file list [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.152659094Z ./ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.152768877Z aggregated-discovery-api.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.153135557Z aggregated-discovery-apis.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.154008991Z event-filter.html [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.156263253Z timestamp [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.156348735Z version [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165009084Z cluster-scoped-resources/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165041704Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165047685Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165101806Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165215599Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165314712Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165378094Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165425365Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165507657Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.16559124Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165706712Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165758814Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.165889858Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.16596174Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.166408042Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.166450923Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.166628908Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.166835684Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.16705688Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.167185643Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.170497554Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.172034727Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.17216588Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.172301634Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.172423687Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.172549881Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.173051825Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.173181748Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.173360903Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.173473716Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.1743362Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.174571756Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.1746971Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.174869444Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.176053037Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.176267843Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.176431627Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.176594012Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.17690624Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.177044474Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.17724364Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.177501337Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.178234927Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.17834516Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.178457413Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.178568236Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.178672499Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.178778032Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.178959247Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.179153312Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.180013126Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.180138479Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.180267503Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.182164215Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterservingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.18271667Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.182905925Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.18308238Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.183296936Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.183561573Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.184015606Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.184127689Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.184249032Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.184355625Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.184466158Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.184583731Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.184687894Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.184867179Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.185012143Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.185129706Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.185368473Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.185555228Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.185645491Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.186280368Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.186382801Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.186504144Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.18669476Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.18745461Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.187570864Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.187752479Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.187865542Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.188057157Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.18817213Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.188343155Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.188472729Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.188605522Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.188724595Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.188856199Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.188988453Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.189102156Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.18924344Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.189376343Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.189507507Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.189655621Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.190042922Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.190210776Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.192797737Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.192957872Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.193767024Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.193913678Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.194058892Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.194148425Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.194322529Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.194467143Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.19471462Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.194872734Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.194985348Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.195148312Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.195271475Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.195389689Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.195534723Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.195659436Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.199514182Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.199983395Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.2005295Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.200716335Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.20089848Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.201042384Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.201174807Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.201282621Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.201385193Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.202235487Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.202554376Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.207019658Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.207145272Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.207286826Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.207414249Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.207535782Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.207644516Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.207759249Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.210442913Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.213567098Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.2194341Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.225709182Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.225913448Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.226279968Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.226464933Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.226650958Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.226786622Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.227026988Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.227344097Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.227480261Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.227612854Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.227871281Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.228031546Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.228201281Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.228325884Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.228457358Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.228623262Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.228816527Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.228967052Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.229132076Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.22928152Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.229444165Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.229576859Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.229739973Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.229920338Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.230091693Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.230442272Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.230605707Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.230919595Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.231065539Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.231217273Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.231350027Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.233783924Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.233966489Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.234118703Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.234252687Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.234384041Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.234571516Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.23472124Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.234933906Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.23509755Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.23689344Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.237101775Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.237251219Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.237434804Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.237579328Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.237798194Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.238119543Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.238700129Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.239039659Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.239193633Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.239329567Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.23946314Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.240313644Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.24053446Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.240679024Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.2423832Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.242563396Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.242816132Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.242912375Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.243133421Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.24344933Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.243623035Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.24381905Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.244312544Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.244433657Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.244581841Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.244789347Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.244998703Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.245165657Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.245329021Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.245503646Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.245944578Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.24601351Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.246142654Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.246244127Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.24635786Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.246458383Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.246647318Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.246774981Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.246909945Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.247026958Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.247124281Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.247242304Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.247352557Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.24746504Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.247578983Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.247689456Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.247800539Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.247981624Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.248098157Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.248209131Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.248318404Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.248427477Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.24853958Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.248651093Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.248763296Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.248899379Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.249020883Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.249132766Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.249250409Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.249364862Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.249481295Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.249592659Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.249703362Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.249810955Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.249951848Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.250078832Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.250179735Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.250296788Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.250413771Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.250536795Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.250672608Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.250784691Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.250925245Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.251032198Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.251135391Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.251255724Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.251367027Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.251490501Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.251631865Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.251757148Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.251893342Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.252013405Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.252135199Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.252255852Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.252369725Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.252488878Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.252602872Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.252713555Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.252824667Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.252969311Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.253077194Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.253197608Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.253318211Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.253433544Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.253556578Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.253692011Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.253809305Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.253947688Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.254073502Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.254195015Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.254311378Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.254425391Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.254541095Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.254654348Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.254771281Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.254906655Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.255026958Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.255138791Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.255254984Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.255372997Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.255491781Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.255601444Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.255737408Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.255853891Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.256001865Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.256122038Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.256243001Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.256355805Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.256472298Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.256585521Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.256702264Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.256815957Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.257009302Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.257106505Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.257221768Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.257332811Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.257475455Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.257570748Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.257687341Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.257808324Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.257949299Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.258066112Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.258185985Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.258312538Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.258422521Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.258542595Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.258654708Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.258870154Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.258917825Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.259130921Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.259293425Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.259404188Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.259512242Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.259897932Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.260038926Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.260299003Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.260413596Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.26054668Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.260653753Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.260759546Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.260884339Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.261006623Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.261111195Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.261233019Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.261342282Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.261462235Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.261569538Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.261681371Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.261788354Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.261937618Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.262051891Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.262185705Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.262214396Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.26236212Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.262480353Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.262588286Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.262707689Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.262824882Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.263020018Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.263227004Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.263317306Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.263411559Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.263512792Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.263607374Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.263777599Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.263913152Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264049156Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264147049Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264248592Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264346234Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264435367Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264534949Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264625402Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264691244Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264741775Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264800477Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264859809Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264933391Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.264981112Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265086835Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265142286Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265204738Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265233449Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.26529267Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265323601Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265382743Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265416864Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265462175Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265567818Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.26566058Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265710122Z cluster-scoped-resources/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265719522Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265777504Z cluster-scoped-resources/core/nodes/ip-10-0-130-79.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.265958389Z cluster-scoped-resources/core/nodes/ip-10-0-141-13.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.266110593Z cluster-scoped-resources/core/nodes/ip-10-0-142-209.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.266232156Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.266258757Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.266315518Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.266410731Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.266496363Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.266583146Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.266667998Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.266754861Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.266927945Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.267028778Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.267124551Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.267224894Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.267310466Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.267406819Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.267500981Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.267599504Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.267701857Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.267789029Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.267901202Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.267991215Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.268086407Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.26817356Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.268261222Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.268354925Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.268390706Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.268497898Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.268612692Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.268715655Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.268795687Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.26890433Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269062854Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269183857Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.26927685Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269330631Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269351512Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269403153Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269475905Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269484996Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269514366Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269594869Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.26963819Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269693011Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269703322Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269733732Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.269825105Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.26998965Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270056641Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270071902Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270121793Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270242416Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270250007Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270299878Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.2703741Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270410981Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270531784Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270585566Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270657468Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270697069Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270771281Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270808972Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.270973946Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271018378Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271070979Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27111068Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271147881Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271198012Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271234673Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271287465Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271345077Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271399488Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27147308Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271522512Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271591283Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271636165Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271706717Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271746328Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271810769Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271861071Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.271997005Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.272004415Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.272052826Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.272145689Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27219077Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.272292253Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.272393336Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27257325Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.272699164Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.272829928Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.273009862Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.273048463Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.273111075Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.273152066Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.273249789Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.273332021Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.273486335Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.273631079Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.273726622Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.273818875Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.273936328Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.2740265Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.274118313Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.274205905Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.274298588Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27438394Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.274470253Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.274564655Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.274648407Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27474285Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.274829692Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.274941926Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.275029608Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27511398Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.275232394Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.275282315Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.275378838Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27545812Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.275544692Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.275689606Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.275784429Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.275931743Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.276049786Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.276137248Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.276231881Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.276332214Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.276428436Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.276516669Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.276609631Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.276702074Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.276803567Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.2769176Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.277004882Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.277098085Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.277182217Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27727671Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.277369512Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.277451135Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.277537847Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.277628279Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.277717122Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.277801334Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.277916257Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27800399Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278111953Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278171324Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278188685Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278256697Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278326739Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278334649Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27838009Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278444252Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278453112Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278486153Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278574566Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278621717Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-130-79.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278709449Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-141-13.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278790161Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-142-209.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278852593Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278893294Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.278981767Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279031268Z host_service_logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.27908376Z host_service_logs/masters/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279150211Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279232944Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279297395Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279366637Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279490391Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279594894Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279663375Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279738658Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279804689Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279901072Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279974794Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.279995185Z ingress_controllers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.280001655Z ingress_controllers/default/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.280005975Z ingress_controllers/default/router-default-657b5f585b-96g68/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.280060346Z ingress_controllers/default/router-default-657b5f585b-96g68/haproxy.config [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.28019162Z insights-data/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.28020096Z istio/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.280245371Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.280376295Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.280899879Z istio/event-filter.html [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.281054824Z istio/timestamp [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.281157166Z istio/cluster-scoped-resources/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.281168037Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.281174597Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.281237579Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.281320331Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.281355692Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.281431014Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.281478395Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.281532477Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.281694091Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.282614446Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.282812242Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.283019938Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.283399508Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.283573843Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.283985844Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.284855598Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.285004192Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.285128536Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.285239819Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.285357902Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.285489765Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.28563724Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.285812514Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.286113293Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.286290018Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.286677418Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.286812432Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.286976766Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.28710973Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.28712189Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.287165342Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.287242434Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.287256384Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.287273365Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.287383548Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.287531982Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.287616884Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.287681076Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.287856891Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288022365Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288096087Z istio/namespaces/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288105268Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288114808Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/gateway.networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288120008Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288160459Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/gateway.networking.k8s.io/httproutes/fc-concurrency-test-kserve-route.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288260392Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288267772Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288308573Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.k8s.io/inferencepools/fc-concurrency-test-inference-pool.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288386185Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.x-k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288393465Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288432346Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/inference.networking.x-k8s.io/inferencepools/fc-concurrency-test-inference-pool.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288488768Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/networking.istio.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288500758Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/networking.istio.io/destinationrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.2885525Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/networking.istio.io/destinationrules/fc-concurrency-test-kserve-scheduler.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288649852Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/networking.istio.io/destinationrules/fc-concurrency-test-kserve-shadow-svc.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288736065Z istio/namespaces/e2e-test-flow-control-smoke-1e98f46d/networking.istio.io/destinationrules/fc-concurrency-test-kserve-workload-svc.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288863488Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288972481Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/gateway.networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.288981392Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289029183Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/gateway.networking.k8s.io/httproutes/scheduler-v06-threshold-migration-test-kserve-route.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289128226Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/inference.networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289140206Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289191577Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/inference.networking.k8s.io/inferencepools/scheduler-v06-threshold-migration-test-inference-pool.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289244349Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/inference.networking.x-k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289251399Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.28928933Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/inference.networking.x-k8s.io/inferencepools/scheduler-v06-threshold-migration-test-inference-pool.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289352022Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/networking.istio.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289362642Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/networking.istio.io/destinationrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289422504Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/networking.istio.io/destinationrules/scheduler-v06-threshold-migration-test-kserve-scheduler.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289514596Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/networking.istio.io/destinationrules/scheduler-v06-threshold-migration-test-kserve-shadow-svc.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289667391Z istio/namespaces/e2e-test-llm-inference-service-40b85f95/networking.istio.io/destinationrules/scheduler-v06-threshold-migration-test-kserve-workload-svc.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289735912Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289745653Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/gateway.networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289750033Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289795014Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/gateway.networking.k8s.io/httproutes/scheduler-v06-pd-migration-test-kserve-route.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289905877Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/inference.networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289916607Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.289967939Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/inference.networking.k8s.io/inferencepools/scheduler-v06-pd-migration-test-inference-pool.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290040331Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/inference.networking.x-k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290053691Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290102932Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/inference.networking.x-k8s.io/inferencepools/scheduler-v06-pd-migration-test-inference-pool.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290158334Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/networking.istio.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290172634Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/networking.istio.io/destinationrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290216575Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-scheduler.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290306748Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-shadow-svc.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290452132Z istio/namespaces/e2e-test-llm-inference-service-9c386aa9/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-workload-svc.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290571595Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290614137Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290695839Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.29072937Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290810062Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290858753Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.290969526Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.291191462Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.291429379Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.291494011Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.291544932Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.291629545Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.291653745Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.291759648Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.291805979Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.291859661Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.291976674Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.292054086Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.292096877Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.292529999Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.292627942Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.292870388Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.292972331Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.293218808Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.293341592Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.293554517Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.2936379Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.293673551Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.293777604Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.293785074Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.293818975Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.293920817Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.293963818Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.29400996Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294069441Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294158614Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294243306Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294288827Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294329199Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294385Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.29439254Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294431811Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294524824Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294707189Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294814502Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294868383Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294930825Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294939065Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.294986466Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/istiod-openshift-gateway-5677ccb49b-zhst7.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.29509134Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.29509951Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.29510619Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.295162761Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.298772941Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.298853343Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.298901514Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.298950186Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.299077709Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.299086839Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.29910249Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.29912676Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.299387278Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.29948429Z istio/namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.299536902Z istio/namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router-default-657b5f585b-96g68.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.299644145Z istio/namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.299652055Z istio/namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.299660135Z istio/namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.299709036Z istio/namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.29983011Z istio/namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.299933213Z istio/namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.299951823Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.300003575Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.300050326Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.300100687Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.300147619Z machine_config_ondisk/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.300155049Z machine_config_termination_logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.300159549Z monitoring/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.300163649Z monitoring/alertmanager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.30020103Z monitoring/alertmanager/status.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.300296612Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.300353864Z monitoring/prometheus/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.300392865Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.300487228Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.30055383Z monitoring/prometheus/rules.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.301343951Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.301391653Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.301437424Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.302319738Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.30237368Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.302423071Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.302512853Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.302574865Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.302668118Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.302719539Z monitoring/prometheus/status/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.30274708Z monitoring/prometheus/status/config.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.303224953Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.303295835Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.303380127Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.303595473Z namespaces/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.303605923Z namespaces/cert-manager-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.303610194Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.303613974Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.303680326Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.303755908Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.303779678Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.3038215Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.304047786Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.304261762Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.304414796Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.304622502Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.304900989Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.304955011Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-xx68k.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.305117885Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.305158166Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.305219228Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.305253989Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.305319571Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.305358962Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.305436604Z namespaces/cert-manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.305443914Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.305457525Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.305504976Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.30565896Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.305862906Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.30601439Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.306231976Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.306528164Z namespaces/default/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.306577305Z namespaces/default/default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.306643497Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.306685198Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.30675529Z namespaces/default/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.306797441Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.306907374Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.306995087Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307072049Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.30712701Z namespaces/default/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307148181Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307200762Z namespaces/default/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307243974Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307327876Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307388667Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307433749Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307517951Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307599193Z namespaces/default/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307637134Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307764238Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.307867421Z namespaces/default/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.308060356Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.308139278Z namespaces/default/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.308220961Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.308353084Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.308527039Z namespaces/default/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.308594541Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.308636792Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.308698274Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.308740435Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.308790906Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.308830837Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.30893398Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.309030243Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.309055093Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.309112155Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.309140286Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.309200237Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.309249799Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.309262129Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.30931248Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.309472415Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.30964443Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.309783544Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.31000607Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310218055Z namespaces/default/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310252006Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310302788Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310350789Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310418511Z namespaces/e2e-test-flow-control-smoke-1e98f46d/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310434091Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310443411Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310482193Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/podmonitors/fc-concurrency-test-kserve-llmisvc-engine-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310582966Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/podmonitors/fc-concurrency-test-kserve-llmisvc-engine.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310640577Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310660688Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.31075193Z namespaces/e2e-test-flow-control-smoke-1e98f46d/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310794611Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310856253Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.310880234Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.311051408Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.311225323Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.311371517Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.311569342Z namespaces/e2e-test-flow-control-smoke-1e98f46d/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.311870371Z namespaces/e2e-test-llm-inference-service-40b85f95/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.311882261Z namespaces/e2e-test-llm-inference-service-40b85f95/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.311890771Z namespaces/e2e-test-llm-inference-service-40b85f95/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.311936883Z namespaces/e2e-test-llm-inference-service-40b85f95/monitoring.coreos.com/podmonitors/s1c85672a3c61368a730e747fe1b48c7f-kserve-llmisvc-engine-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.312044066Z namespaces/e2e-test-llm-inference-service-40b85f95/monitoring.coreos.com/podmonitors/scheduler-v06-threshold-migration-test-kserve-llmisvc-engine.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.312168609Z namespaces/e2e-test-llm-inference-service-40b85f95/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.31219506Z namespaces/e2e-test-llm-inference-service-40b85f95/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.312283252Z namespaces/e2e-test-llm-inference-service-40b85f95/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.312340914Z namespaces/e2e-test-llm-inference-service-40b85f95/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.312350614Z namespaces/e2e-test-llm-inference-service-40b85f95/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.312413636Z namespaces/e2e-test-llm-inference-service-40b85f95/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.31256761Z namespaces/e2e-test-llm-inference-service-40b85f95/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.312754445Z namespaces/e2e-test-llm-inference-service-40b85f95/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.312958831Z namespaces/e2e-test-llm-inference-service-40b85f95/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313167366Z namespaces/e2e-test-llm-inference-service-40b85f95/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313348671Z namespaces/e2e-test-llm-inference-service-9c386aa9/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313358042Z namespaces/e2e-test-llm-inference-service-9c386aa9/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313362962Z namespaces/e2e-test-llm-inference-service-9c386aa9/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313375532Z namespaces/e2e-test-llm-inference-service-9c386aa9/monitoring.coreos.com/podmonitors/scheduler-v06-pd-migration-test-kserve-llmisvc-engine-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313492646Z namespaces/e2e-test-llm-inference-service-9c386aa9/monitoring.coreos.com/podmonitors/scheduler-v06-pd-migration-test-kserve-llmisvc-engine.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313585478Z namespaces/e2e-test-llm-inference-service-9c386aa9/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313629669Z namespaces/e2e-test-llm-inference-service-9c386aa9/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313725892Z namespaces/e2e-test-llm-inference-service-9c386aa9/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313775083Z namespaces/e2e-test-llm-inference-service-9c386aa9/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313791144Z namespaces/e2e-test-llm-inference-service-9c386aa9/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.313862056Z namespaces/e2e-test-llm-inference-service-9c386aa9/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.314101222Z namespaces/e2e-test-llm-inference-service-9c386aa9/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.314213005Z namespaces/e2e-test-llm-inference-service-9c386aa9/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.314405311Z namespaces/e2e-test-llm-inference-service-9c386aa9/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.314581195Z namespaces/e2e-test-llm-inference-service-9c386aa9/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.3147571Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.314765401Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.314777961Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.314835472Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.315009087Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.315201493Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.315378177Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.315618434Z namespaces/e2e-test-llm-tls-resources-e4cad0c3/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.315813509Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.31582644Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.31585301Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.315924302Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.316072746Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.316251421Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.316385565Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.316593201Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.316823557Z namespaces/kserve/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.316890559Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.316960091Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.317016462Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.317085744Z namespaces/kserve/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.317122575Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.317220498Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.317449824Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.317617529Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.31766719Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.317701471Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.317753283Z namespaces/kserve/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.317794114Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.317903687Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.317976649Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.31801181Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.318119473Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.318171254Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.318185304Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.318199515Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.318309178Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.31839531Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.318505073Z namespaces/kserve/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.318553665Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.318793271Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.318935995Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.319318535Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.319400668Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.319682185Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.319806389Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.320095307Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.32019767Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.32021823Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.320347684Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.320387675Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.320455467Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.320502678Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.320594281Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.320669353Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.320743745Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.320779806Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.320861788Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.3209574Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.321043863Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.321095724Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.321140156Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.321199217Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.321209747Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.321266889Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.321420553Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.321601198Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.321749482Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.321962428Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322235956Z namespaces/kserve/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322293947Z namespaces/kserve/pods/kserve-controller-manager-575575dfd7-nrvzj/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322337538Z namespaces/kserve/pods/kserve-controller-manager-575575dfd7-nrvzj/kserve-controller-manager-575575dfd7-nrvzj.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322457222Z namespaces/kserve/pods/kserve-controller-manager-575575dfd7-nrvzj/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322465582Z namespaces/kserve/pods/kserve-controller-manager-575575dfd7-nrvzj/manager/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322476922Z namespaces/kserve/pods/kserve-controller-manager-575575dfd7-nrvzj/manager/manager/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322514423Z namespaces/kserve/pods/kserve-controller-manager-575575dfd7-nrvzj/manager/manager/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322639127Z namespaces/kserve/pods/kserve-controller-manager-575575dfd7-nrvzj/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322714979Z namespaces/kserve/pods/kserve-controller-manager-575575dfd7-nrvzj/manager/manager/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.32276009Z namespaces/kserve/pods/llmisvc-controller-manager-7774f8b66-9w5pn/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322803921Z namespaces/kserve/pods/llmisvc-controller-manager-7774f8b66-9w5pn/llmisvc-controller-manager-7774f8b66-9w5pn.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322920434Z namespaces/kserve/pods/llmisvc-controller-manager-7774f8b66-9w5pn/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322935475Z namespaces/kserve/pods/llmisvc-controller-manager-7774f8b66-9w5pn/manager/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322944245Z namespaces/kserve/pods/llmisvc-controller-manager-7774f8b66-9w5pn/manager/manager/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.322979996Z namespaces/kserve/pods/llmisvc-controller-manager-7774f8b66-9w5pn/manager/manager/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376380994Z namespaces/kserve/pods/llmisvc-controller-manager-7774f8b66-9w5pn/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376420025Z namespaces/kserve/pods/llmisvc-controller-manager-7774f8b66-9w5pn/manager/manager/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376485447Z namespaces/kserve/pods/model-serving-api-7b954ffc95-x7szn/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376538989Z namespaces/kserve/pods/model-serving-api-7b954ffc95-x7szn/model-serving-api-7b954ffc95-x7szn.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376637361Z namespaces/kserve/pods/model-serving-api-7b954ffc95-x7szn/server/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376646442Z namespaces/kserve/pods/model-serving-api-7b954ffc95-x7szn/server/server/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376655322Z namespaces/kserve/pods/model-serving-api-7b954ffc95-x7szn/server/server/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376705873Z namespaces/kserve/pods/model-serving-api-7b954ffc95-x7szn/server/server/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376804616Z namespaces/kserve/pods/model-serving-api-7b954ffc95-x7szn/server/server/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376902649Z namespaces/kserve/pods/model-serving-api-7b954ffc95-x7szn/server/server/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376931279Z namespaces/kserve/pods/odh-model-controller-7b966d7984-kv7l2/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.376986451Z namespaces/kserve/pods/odh-model-controller-7b966d7984-kv7l2/odh-model-controller-7b966d7984-kv7l2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.377072593Z namespaces/kserve/pods/odh-model-controller-7b966d7984-kv7l2/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.377094504Z namespaces/kserve/pods/odh-model-controller-7b966d7984-kv7l2/manager/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.377102594Z namespaces/kserve/pods/odh-model-controller-7b966d7984-kv7l2/manager/manager/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.377139795Z namespaces/kserve/pods/odh-model-controller-7b966d7984-kv7l2/manager/manager/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.401915626Z namespaces/kserve/pods/odh-model-controller-7b966d7984-kv7l2/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.401985758Z namespaces/kserve/pods/odh-model-controller-7b966d7984-kv7l2/manager/manager/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402012839Z namespaces/kserve/pods/s3-init-9jbt8/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402077801Z namespaces/kserve/pods/s3-init-9jbt8/s3-init-9jbt8.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402167603Z namespaces/kserve/pods/s3-init-9jbt8/download-hf-model/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402180524Z namespaces/kserve/pods/s3-init-9jbt8/download-hf-model/download-hf-model/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402185324Z namespaces/kserve/pods/s3-init-9jbt8/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402227555Z namespaces/kserve/pods/s3-init-9jbt8/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402311597Z namespaces/kserve/pods/s3-init-9jbt8/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402397739Z namespaces/kserve/pods/s3-init-9jbt8/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.40241704Z namespaces/kserve/pods/s3-init-9jbt8/s3-init/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.40242238Z namespaces/kserve/pods/s3-init-9jbt8/s3-init/s3-init/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.40243676Z namespaces/kserve/pods/s3-init-9jbt8/s3-init/s3-init/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402497902Z namespaces/kserve/pods/s3-init-9jbt8/s3-init/s3-init/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.4027614Z namespaces/kserve/pods/s3-init-9jbt8/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402855362Z namespaces/kserve/pods/s3-init-9jbt8/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402879173Z namespaces/kserve/pods/seaweedfs-64568bcd49-8fqmw/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.402938324Z namespaces/kserve/pods/seaweedfs-64568bcd49-8fqmw/seaweedfs-64568bcd49-8fqmw.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403009746Z namespaces/kserve/pods/seaweedfs-64568bcd49-8fqmw/seaweedfs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403025017Z namespaces/kserve/pods/seaweedfs-64568bcd49-8fqmw/seaweedfs/seaweedfs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403037317Z namespaces/kserve/pods/seaweedfs-64568bcd49-8fqmw/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403085849Z namespaces/kserve/pods/seaweedfs-64568bcd49-8fqmw/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403296534Z namespaces/kserve/pods/seaweedfs-64568bcd49-8fqmw/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403362316Z namespaces/kserve/pods/seaweedfs-64568bcd49-8fqmw/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403402137Z namespaces/kserve/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403444838Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.40351585Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403540401Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403638974Z namespaces/kuadrant-system/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403686965Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403759417Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403807978Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.40388319Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.403920611Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.404009454Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.404292402Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.40458503Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.404622971Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.404660722Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.404733174Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.404774915Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.404880218Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.404924709Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.40497867Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.405061653Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.405102204Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.405108594Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.405158955Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.405245318Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.40533937Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.405423153Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.405479234Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.405524936Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.40569149Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.405818593Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.406263296Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.406346488Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.406751739Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.406906363Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.407121069Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.407233603Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.407279374Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.407403147Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.407437018Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.40751189Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.407532011Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.407638484Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.407713776Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.407769437Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.407805448Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.40788182Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.407923382Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.408004724Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.408018744Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.408058765Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.408266431Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.408473137Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.408628511Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.408858177Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.409077733Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.409115864Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-vn6wj.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.409423883Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.409453373Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.409551266Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.409635989Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.409724881Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.409787893Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.409833104Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.409918636Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.409958458Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.410087911Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.410262776Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.410358629Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.410443251Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.410452711Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.410505182Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino-6d9898bd99-7df2d.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.410571544Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.410580224Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/authorino/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.410592075Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/authorino/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.410659197Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/authorino/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.422856392Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.422926984Z namespaces/kuadrant-system/pods/authorino-6d9898bd99-7df2d/authorino/authorino/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.422981365Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423036167Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/authorino-operator-6fc8f864f4-4srxn.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.42314582Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.42315546Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.42316062Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/manager/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423208632Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/manager/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423376016Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423446688Z namespaces/kuadrant-system/pods/authorino-operator-6fc8f864f4-4srxn/manager/manager/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.4235007Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423542581Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/dns-operator-controller-manager-65dc8f76f5-4sgbx.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423652124Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423660964Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423668364Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/manager/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423715516Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/manager/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423831479Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423927231Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65dc8f76f5-4sgbx/manager/manager/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.423962602Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424007393Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin-fbdd4f445-csmlv.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424081666Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424090916Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424095686Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424140047Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424205779Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424268901Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-fbdd4f445-csmlv/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424301962Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424347373Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/kuadrant-operator-controller-manager-6d4cb79c-vfzqt.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424455866Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424464476Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424468746Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/manager/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.424503677Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/manager/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.437876665Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.437961187Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-6d4cb79c-vfzqt/manager/manager/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438001258Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.43806583Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador-limitador-84994f7ffc-2z8v7.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438160823Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438178273Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/limitador/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438184283Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/limitador/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438223115Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/limitador/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438319487Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438381629Z namespaces/kuadrant-system/pods/limitador-limitador-84994f7ffc-2z8v7/limitador/limitador/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.4384355Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438471991Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438580354Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438592265Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438597955Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/manager/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438624745Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/manager/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.43879479Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438857122Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-5ffb4b4ccb-tbv6j/manager/manager/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438908223Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.438966355Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.439011676Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.439045337Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.439082938Z namespaces/kube-node-lease/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.439090598Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.439098998Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.43917285Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-130-79.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.439250703Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-141-13.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.439340825Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-142-209.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.439394767Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.439420597Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.439458638Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.439655824Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.43988665Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.440041144Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.44023483Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.440445585Z namespaces/kube-public/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.440457576Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.440467976Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.440509367Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.440662521Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.440881577Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.441023611Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.441210257Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.441451473Z namespaces/kube-system/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.441498764Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.441562006Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.441604877Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.441655329Z namespaces/kube-system/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.4416993Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.441822823Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.441940306Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442037309Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442107641Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442156163Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442206464Z namespaces/kube-system/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442243505Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442340548Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442361108Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.44242693Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442505622Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442548243Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442566674Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442623045Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-buow5viav24lqut2nmbhqvfdr4.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442710758Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.44280963Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.442919164Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.443066297Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.44313883Z namespaces/kube-system/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.443187381Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.443339125Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.443426827Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.443684445Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.443763817Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.444061535Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.444175718Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.444594099Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.444661951Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.444705733Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.444761894Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.444799245Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.444852937Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.444914418Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.445002691Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.445092943Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.445140705Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.445199866Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.445215857Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.445266848Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.445309099Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.44534395Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.445386681Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.445541396Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.44571738Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.445854184Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.446099441Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.446313437Z namespaces/kube-system/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.446323017Z namespaces/kube-system/pods/global-pull-secret-syncer-5nrdf/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.446372668Z namespaces/kube-system/pods/global-pull-secret-syncer-5nrdf/global-pull-secret-syncer-5nrdf.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.446451741Z namespaces/kube-system/pods/global-pull-secret-syncer-5nrdf/global-pull-secret-syncer/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.446459281Z namespaces/kube-system/pods/global-pull-secret-syncer-5nrdf/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.446463481Z namespaces/kube-system/pods/global-pull-secret-syncer-5nrdf/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.446503782Z namespaces/kube-system/pods/global-pull-secret-syncer-5nrdf/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.446860942Z namespaces/kube-system/pods/global-pull-secret-syncer-5nrdf/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.446941194Z namespaces/kube-system/pods/global-pull-secret-syncer-5nrdf/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.446960984Z namespaces/kube-system/pods/global-pull-secret-syncer-gddqh/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447011476Z namespaces/kube-system/pods/global-pull-secret-syncer-gddqh/global-pull-secret-syncer-gddqh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447090448Z namespaces/kube-system/pods/global-pull-secret-syncer-gddqh/global-pull-secret-syncer/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447098808Z namespaces/kube-system/pods/global-pull-secret-syncer-gddqh/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447103259Z namespaces/kube-system/pods/global-pull-secret-syncer-gddqh/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447141809Z namespaces/kube-system/pods/global-pull-secret-syncer-gddqh/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447481839Z namespaces/kube-system/pods/global-pull-secret-syncer-gddqh/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447550771Z namespaces/kube-system/pods/global-pull-secret-syncer-gddqh/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447581932Z namespaces/kube-system/pods/global-pull-secret-syncer-sdwt8/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447627273Z namespaces/kube-system/pods/global-pull-secret-syncer-sdwt8/global-pull-secret-syncer-sdwt8.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447699815Z namespaces/kube-system/pods/global-pull-secret-syncer-sdwt8/global-pull-secret-syncer/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447707125Z namespaces/kube-system/pods/global-pull-secret-syncer-sdwt8/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447717995Z namespaces/kube-system/pods/global-pull-secret-syncer-sdwt8/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.447775767Z namespaces/kube-system/pods/global-pull-secret-syncer-sdwt8/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.448155657Z namespaces/kube-system/pods/global-pull-secret-syncer-sdwt8/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.448224769Z namespaces/kube-system/pods/global-pull-secret-syncer-sdwt8/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.448275461Z namespaces/kube-system/pods/konnectivity-agent-485xb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.448307322Z namespaces/kube-system/pods/konnectivity-agent-485xb/konnectivity-agent-485xb.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.448418235Z namespaces/kube-system/pods/konnectivity-agent-485xb/konnectivity-agent/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.448425325Z namespaces/kube-system/pods/konnectivity-agent-485xb/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.448432165Z namespaces/kube-system/pods/konnectivity-agent-485xb/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.448484626Z namespaces/kube-system/pods/konnectivity-agent-485xb/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.449529075Z namespaces/kube-system/pods/konnectivity-agent-485xb/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.449613617Z namespaces/kube-system/pods/konnectivity-agent-485xb/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.449664699Z namespaces/kube-system/pods/konnectivity-agent-976mf/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.44970404Z namespaces/kube-system/pods/konnectivity-agent-976mf/konnectivity-agent-976mf.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.449782612Z namespaces/kube-system/pods/konnectivity-agent-976mf/konnectivity-agent/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.449793782Z namespaces/kube-system/pods/konnectivity-agent-976mf/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.449798403Z namespaces/kube-system/pods/konnectivity-agent-976mf/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.449820653Z namespaces/kube-system/pods/konnectivity-agent-976mf/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.450867132Z namespaces/kube-system/pods/konnectivity-agent-976mf/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.450950234Z namespaces/kube-system/pods/konnectivity-agent-976mf/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.450982285Z namespaces/kube-system/pods/konnectivity-agent-tpwpt/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.451027046Z namespaces/kube-system/pods/konnectivity-agent-tpwpt/konnectivity-agent-tpwpt.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.451108809Z namespaces/kube-system/pods/konnectivity-agent-tpwpt/konnectivity-agent/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.451117699Z namespaces/kube-system/pods/konnectivity-agent-tpwpt/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.451125349Z namespaces/kube-system/pods/konnectivity-agent-tpwpt/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.451174011Z namespaces/kube-system/pods/konnectivity-agent-tpwpt/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.452361783Z namespaces/kube-system/pods/konnectivity-agent-tpwpt/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.452433855Z namespaces/kube-system/pods/konnectivity-agent-tpwpt/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.452487496Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.452527608Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45261253Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal/haproxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.4526271Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.452631851Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.452681242Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453064292Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453131834Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453197446Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453236067Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45332391Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal/haproxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.4533327Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45333953Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453375051Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453661189Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453730911Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-13.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453771912Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453816003Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453910776Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal/haproxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453922276Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453926986Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.453948197Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.454197274Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.454267256Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.454298766Z namespaces/kube-system/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.454342618Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.454396049Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45441346Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45444056Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.454540503Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.454608585Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.454640486Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.454703117Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.454733488Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45478958Z namespaces/open-cluster-management-0bbc98b4-c464-4493-b02e-7e9bfa164/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45479675Z namespaces/open-cluster-management-0bbc98b4-c464-4493-b02e-7e9bfa164/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45480542Z namespaces/open-cluster-management-0bbc98b4-c464-4493-b02e-7e9bfa164/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.454876802Z namespaces/open-cluster-management-0bbc98b4-c464-4493-b02e-7e9bfa164/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.455093228Z namespaces/open-cluster-management-0bbc98b4-c464-4493-b02e-7e9bfa164/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.455291154Z namespaces/open-cluster-management-0bbc98b4-c464-4493-b02e-7e9bfa164/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.455437868Z namespaces/open-cluster-management-0bbc98b4-c464-4493-b02e-7e9bfa164/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.455646003Z namespaces/open-cluster-management-0bbc98b4-c464-4493-b02e-7e9bfa164/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45588879Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.455903481Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.455908071Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.455945352Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.456039884Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.456091686Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.456111076Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.456150597Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.456306881Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.456485826Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45662287Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.456813755Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457061832Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457106574Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457170155Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457222367Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457294099Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45733954Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457425732Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457504154Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457593397Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457635658Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457664229Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457728161Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457770422Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457861234Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457938786Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.457959857Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45807233Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.458127252Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.458154122Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.459082778Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.45916705Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.459251953Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.459338325Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.459423267Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.459581552Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.459768187Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460018454Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460053104Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460129967Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460148907Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460218589Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460267621Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460346213Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460424345Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460494117Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460513937Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.46059954Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460636661Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460696042Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460744664Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460808525Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460817216Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.460852226Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.461092743Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.461364851Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.461500984Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.46170667Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.461909646Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.461955137Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.461999928Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.4620551Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462149542Z namespaces/openshift-apiserver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462205594Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462273036Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462322527Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462370218Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.46241948Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462498092Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462586474Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462670836Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462714158Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462746789Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.46281236Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462861702Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.462962104Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.463008316Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.463063107Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.463145829Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.463219891Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.463252032Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.463373606Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.463451238Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.46353604Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.463612922Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.463695775Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.463823308Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.463994713Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464043634Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464098596Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464159977Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464207979Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464265Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464308312Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464400324Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464482676Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464538498Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464575629Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464642551Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464680222Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464733613Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464743853Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.464798445Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.4649806Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.465159245Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.465297919Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.465508045Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.465683249Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.465730451Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.465784612Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.465833613Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.465899055Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.465916986Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.465926286Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.465972637Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.466041349Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.466250105Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.466291086Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.466462591Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.466635945Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.466776589Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.467008066Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.467192431Z namespaces/openshift-authentication/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.467201611Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.467215841Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.467267413Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.467420837Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.467596962Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.467734496Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.467945152Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.468122716Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.468133607Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.468138567Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.468188258Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.468239409Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.46825762Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.468309891Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.468468506Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.468644881Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.468783735Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.469036691Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.469235677Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.469246587Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.469252057Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.469284368Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.46934252Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.46935332Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.469392291Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.469461433Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.469472734Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.469527455Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.469677459Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.469971277Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.470131312Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.470320107Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.470524342Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.470573904Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.470639635Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.470670346Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.470747318Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.47079104Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.470895673Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.470976525Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471056597Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471103098Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.47115097Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471199261Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471235792Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471338425Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471361475Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471403807Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471495989Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471544731Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471553471Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471602252Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471686544Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471733756Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471856709Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.471947851Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472032944Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472117376Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472195638Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472347253Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472463306Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472504417Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472554258Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472598209Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472648531Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472699922Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472724573Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472811465Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.472863457Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.47297043Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.473052082Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.473102923Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.473141554Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.473192446Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.473221706Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.473280998Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.473296739Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.473315919Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.473468593Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.473648788Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.473789132Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474010308Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474203864Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474252465Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474290506Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474297226Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474347817Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474405529Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.47444727Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474497262Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474544753Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474633735Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474682647Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474749568Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.47479239Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474883932Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.474930633Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475060837Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475143809Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475227172Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475280583Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475316434Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475379036Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475413087Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475504619Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475560921Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475612492Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475690625Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475748586Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475755476Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475811788Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.475929031Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.476015243Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.476137797Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.47623646Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.476281531Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.476388124Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.476467846Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.476816645Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.476915018Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.477117354Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.477258728Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.477373441Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.477407492Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.477462143Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.477790472Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.477850034Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.477957667Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478007648Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.47807926Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478107191Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478200583Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478277185Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478311657Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478357838Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478408659Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.47845539Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478523072Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478547203Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478601465Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478780499Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.478977855Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479114759Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479307764Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479500589Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.47951523Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479553461Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/aws-ebs-csi-driver-node-9bkqb.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479671724Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-driver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479686394Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-driver/csi-driver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479690995Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-driver/csi-driver/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479725495Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479833718Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479929871Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479951762Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-liveness-probe/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479959022Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.479965372Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480028154Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480121036Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480190918Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480225429Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-node-driver-registrar/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480234829Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48024324Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480290951Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480382234Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480465486Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-9bkqb/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480509457Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480548338Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/aws-ebs-csi-driver-node-z7qfj.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48063257Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-driver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480651791Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-driver/csi-driver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480657001Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-driver/csi-driver/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480705222Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480805375Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480901518Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480920688Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-liveness-probe/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480928518Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.480940069Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481001271Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481088593Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481152255Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481190056Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-node-driver-registrar/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481199826Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481207536Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481247907Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48134614Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481434212Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-z7qfj/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481484994Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481539165Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/aws-ebs-csi-driver-node-zg8wt.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481624788Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-driver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481633758Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-driver/csi-driver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481641218Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-driver/csi-driver/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481684459Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481794932Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481887205Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481931376Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-liveness-probe/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481938426Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481944646Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.481985158Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48208937Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482158382Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482216554Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-node-driver-registrar/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482228344Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482233544Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482266975Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482365868Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48244028Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zg8wt/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482483151Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482522142Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482593144Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482603915Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482649146Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482705387Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482757109Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48281153Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482865602Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482933954Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482942644Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.482952574Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.483005555Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.483065547Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.483074228Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.483119809Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48316974Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48318091Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.483228022Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.483512469Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.483695505Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.483824958Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.484071315Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48427134Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.484312662Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.484388954Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.484423434Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.484509227Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.484543198Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.484674712Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.484757964Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.484852446Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48498169Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485023831Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485112533Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485136404Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485200036Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485233777Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485322019Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48535928Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485411532Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485496194Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485539275Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485551245Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485605177Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48569144Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.485731661Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.486711177Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.486826491Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.486999805Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.487115999Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.487307694Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48753721Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.487680804Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.487738505Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.487797117Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.487862149Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.487915201Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.487972362Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488022463Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488090405Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488123036Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488219269Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488304081Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488431265Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488473706Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488547888Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488589339Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488655831Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488714452Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488755873Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488861647Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488906868Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.488937269Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48898908Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.489169965Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.48935901Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.489501944Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.4897369Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.489975397Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.489986827Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q2xgp/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490037149Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q2xgp/tuned-q2xgp.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490145312Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q2xgp/tuned/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490152942Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q2xgp/tuned/tuned/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490164622Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q2xgp/tuned/tuned/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490221254Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q2xgp/tuned/tuned/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490340497Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q2xgp/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490402829Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q2xgp/tuned/tuned/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490468201Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q6bwx/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490507292Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q6bwx/tuned-q6bwx.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490595944Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q6bwx/tuned/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490607604Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q6bwx/tuned/tuned/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490612705Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q6bwx/tuned/tuned/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490676236Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q6bwx/tuned/tuned/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49081252Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q6bwx/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490909593Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-q6bwx/tuned/tuned/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490928263Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-r26z5/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.490982615Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-r26z5/tuned-r26z5.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491063277Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-r26z5/tuned/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491074307Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-r26z5/tuned/tuned/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491080717Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-r26z5/tuned/tuned/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491102458Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-r26z5/tuned/tuned/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491219801Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-r26z5/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491287423Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-r26z5/tuned/tuned/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491337955Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491357565Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491416757Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491452198Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491501489Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49152114Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491568481Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-130-79.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491668934Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-141-13.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491755436Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-142-209.ec2.internal.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491792567Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491865149Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.491961412Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492001973Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492073405Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492107426Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492181728Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49225365Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492337002Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492440445Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492550738Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492602139Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492655881Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492700722Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492732203Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492822825Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492906698Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.492948829Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.493039901Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.493115224Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.493161285Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49333733Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.493419182Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.493546675Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.493624387Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.493741551Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.493979177Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494168672Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494226494Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494266235Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494330237Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494376428Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49444894Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494480661Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494572433Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494649856Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494710787Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494728848Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49478803Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494833411Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494964624Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.494986045Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.495050887Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.495086408Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.495133879Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.495139109Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49518133Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.495433797Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.495609312Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.495749676Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496002943Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496182498Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496189948Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496225629Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator-685d7d68b5-c5vhr.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496323852Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator-watch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496585699Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496594929Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49663696Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496737683Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496807395Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496879177Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496890217Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496898908Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.496940879Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.497610117Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.497691289Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-685d7d68b5-c5vhr/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49773442Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.497777422Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.497858754Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.497915825Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498005868Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49807275Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498129151Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498182263Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498238584Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498284346Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498377438Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498479231Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498580494Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498637205Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498679446Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498738848Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49878936Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498887822Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498929463Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.498975065Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.499059477Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.499105668Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.499115328Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.49916769Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.499252892Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.499335335Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.499380666Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.499493669Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.499576731Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.499980062Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.500066695Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.500183408Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.500326172Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.500476266Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.500538377Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.500590709Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.500658201Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.500736513Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.500815705Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.500881287Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.500959359Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.50100451Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.501090653Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.501169615Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.501233437Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.501266857Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.50134193Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.501381151Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.501479283Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.501519235Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.501567886Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.501599387Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.501693669Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.50170565Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.501757451Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.502006778Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.502207723Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.502343267Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.502553233Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.502734268Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.502743108Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5c4db8bdf9-2rkn8/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.502775119Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5c4db8bdf9-2rkn8/volume-data-source-validator-5c4db8bdf9-2rkn8.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.502857121Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5c4db8bdf9-2rkn8/volume-data-source-validator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.502875522Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5c4db8bdf9-2rkn8/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.502880332Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5c4db8bdf9-2rkn8/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.502919243Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5c4db8bdf9-2rkn8/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503024636Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5c4db8bdf9-2rkn8/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503101798Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5c4db8bdf9-2rkn8/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503304133Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503318544Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503324564Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503330484Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503335265Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503343305Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503378356Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503446388Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503496379Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503582511Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503635553Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503704735Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503725615Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503792357Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503819318Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.503938531Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504016553Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504114616Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504162847Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504216999Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.50427292Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504318632Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504405394Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504464765Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504507707Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504606529Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504660241Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504669661Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504712622Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504799985Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504833106Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.504967969Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505048552Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505155304Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505234777Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505326659Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505449912Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505616177Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505687409Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505743781Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505799532Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505857574Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505914325Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.505969027Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.50606826Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.506152262Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.506186153Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.506243694Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.506298136Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.506336837Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.506392829Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.506402879Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.506467311Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.506614965Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.506789269Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.506947454Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.507147769Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.507343385Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.507375305Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.507451538Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.507482668Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.507584041Z namespaces/openshift-config-managed/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.507616582Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.507819948Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.507878099Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.507961482Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508005933Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508092535Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508171607Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.5082574Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508299001Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508354052Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508390753Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508423074Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508517387Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508577189Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.50860971Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508704332Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508778244Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.508832376Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.511964502Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.512044314Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.512140006Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.512224949Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.512307161Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.512460585Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.512566868Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.51263451Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.512677071Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.512768474Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.512811285Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.512887107Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.512913888Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.51297987Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513027231Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513093913Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513162695Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513241017Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513301828Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.51334578Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513405511Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513458423Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513518234Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513533935Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513577496Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513785142Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.513992067Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.514148602Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.514355777Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.514567353Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.514612554Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.514664856Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.514674206Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.514722587Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.514790779Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.514834501Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.514910103Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.514962044Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.515053067Z namespaces/openshift-config-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.515064577Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.515069287Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.515130929Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.515198611Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.515210551Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.515255962Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.515405596Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.515682714Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.515820217Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.516063864Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.51627935Z namespaces/openshift-config/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.516332002Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.516511187Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.516556258Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.51663074Z namespaces/openshift-config/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.516671901Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.516760494Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.516856416Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.516951219Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.5169988Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517055451Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517114493Z namespaces/openshift-config/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517169515Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517242387Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517303248Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517320129Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517420241Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517461113Z namespaces/openshift-config/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517497404Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517633578Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517705029Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517792022Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517902035Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.517984827Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518160872Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518297156Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518331007Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518390138Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518436869Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518489511Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518538082Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518592964Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518646615Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518699027Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518775199Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.518866881Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.519011735Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.519045346Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.519109548Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.519140999Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.519212981Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.519223611Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.519281033Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.519439617Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.519618242Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.519755596Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.519988592Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.520174387Z namespaces/openshift-config/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.520217578Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.52026662Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.520302411Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.520396754Z namespaces/openshift-console-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.520450775Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.520559408Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.520609829Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.520688782Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.520716072Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.520815255Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.520960119Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521068232Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521113763Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521157514Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521189215Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521241417Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521325489Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.52136136Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521420191Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521503104Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521541645Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521549495Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521594026Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521657248Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.521696509Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.522296896Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.522382948Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.522612294Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.522710487Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.52283101Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.522973884Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523153109Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523270902Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523314764Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523370945Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523414777Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523477018Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523519199Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523602062Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523678184Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523744246Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523789497Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523858039Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.52391228Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.523969472Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.524015553Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.524070455Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.524102845Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.524157107Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.524169177Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.524231489Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.524382113Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.524560868Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.524696172Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.524927988Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.525121003Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.525130104Z namespaces/openshift-console-operator/pods/console-operator-fd4bb46c-q4dqp/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.525160945Z namespaces/openshift-console-operator/pods/console-operator-fd4bb46c-q4dqp/console-operator-fd4bb46c-q4dqp.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.525443792Z namespaces/openshift-console-operator/pods/console-operator-fd4bb46c-q4dqp/console-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.525451762Z namespaces/openshift-console-operator/pods/console-operator-fd4bb46c-q4dqp/console-operator/console-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.525456042Z namespaces/openshift-console-operator/pods/console-operator-fd4bb46c-q4dqp/console-operator/console-operator/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.525499864Z namespaces/openshift-console-operator/pods/console-operator-fd4bb46c-q4dqp/console-operator/console-operator/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.525916825Z namespaces/openshift-console-operator/pods/console-operator-fd4bb46c-q4dqp/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.525972827Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.526013278Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.52608965Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.526131211Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.526200213Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.526210773Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.526216474Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.526235844Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.526405389Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.526583164Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.526706447Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.527014456Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.527240332Z namespaces/openshift-console/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.527273032Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.527610522Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.527632673Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.527723035Z namespaces/openshift-console/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.527759546Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.527862579Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.528044704Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.528336942Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.528375363Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.528440035Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.528485716Z namespaces/openshift-console/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.528535867Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.52861834Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.528676281Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.528720862Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.528789774Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.528855186Z namespaces/openshift-console/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.528934108Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.529514404Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.529597316Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.529853734Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.529956346Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.530137831Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.530269445Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53046661Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.530532312Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.530579063Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.530648435Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.530680916Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.530749218Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.530783079Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.530904152Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.530987505Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.531040386Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.531088257Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.531141829Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53118229Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.531240682Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.531287553Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.531339734Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.531361005Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.531403666Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.5315583Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.531734215Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.531885079Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.532113996Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.532412854Z namespaces/openshift-console/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.532422844Z namespaces/openshift-console/pods/console-67b6fb4769-2gf85/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.532477056Z namespaces/openshift-console/pods/console-67b6fb4769-2gf85/console-67b6fb4769-2gf85.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.532574738Z namespaces/openshift-console/pods/console-67b6fb4769-2gf85/console/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.532582898Z namespaces/openshift-console/pods/console-67b6fb4769-2gf85/console/console/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.532586879Z namespaces/openshift-console/pods/console-67b6fb4769-2gf85/console/console/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53261987Z namespaces/openshift-console/pods/console-67b6fb4769-2gf85/console/console/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.532721182Z namespaces/openshift-console/pods/console-67b6fb4769-2gf85/console/console/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.532798224Z namespaces/openshift-console/pods/console-67b6fb4769-2gf85/console/console/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.532809135Z namespaces/openshift-console/pods/downloads-6594f4c75-bgdjw/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.532893267Z namespaces/openshift-console/pods/downloads-6594f4c75-bgdjw/downloads-6594f4c75-bgdjw.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53299949Z namespaces/openshift-console/pods/downloads-6594f4c75-bgdjw/download-server/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53300769Z namespaces/openshift-console/pods/downloads-6594f4c75-bgdjw/download-server/download-server/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53301169Z namespaces/openshift-console/pods/downloads-6594f4c75-bgdjw/download-server/download-server/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.533018241Z namespaces/openshift-console/pods/downloads-6594f4c75-bgdjw/download-server/download-server/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.533437202Z namespaces/openshift-console/pods/downloads-6594f4c75-bgdjw/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.533504404Z namespaces/openshift-console/pods/downloads-6594f4c75-bgdjw/download-server/download-server/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.533539385Z namespaces/openshift-console/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.533579256Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.533640208Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.533669708Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.533785642Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.533830163Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.533942386Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.533987907Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534011798Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534062439Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534148852Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534226554Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534306116Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534353757Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534390118Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53445277Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534501641Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534581744Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534641095Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534683876Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534761629Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534852521Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.534915533Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535039736Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535119458Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53519883Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535277943Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535348334Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535481888Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535647143Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535696794Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535745355Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535788466Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535822027Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535953441Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.535985032Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.536083115Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.536158317Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.536199198Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.536240709Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.536297111Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.536340112Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.536377823Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.536385423Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.536493686Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53663708Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.536812335Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.536974759Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537168724Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537354909Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537388741Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537423222Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537473073Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537561125Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537594996Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537664878Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537698729Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53773095Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537780551Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537893924Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.537985497Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53809144Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53811149Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538181112Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538216063Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538286445Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538355197Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538398728Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538443099Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538521392Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538571993Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538613174Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538650765Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538742728Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538779459Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.538921343Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.539002585Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.539085077Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.539164439Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.539247832Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.539389025Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53957131Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.539669413Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.539712165Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.539748715Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.539800317Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.539851098Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.53991332Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.539999202Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.540078315Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.540294521Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.540325501Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.540378083Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.540422774Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.540453935Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.540460385Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.540490386Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54064713Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.540825055Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54098884Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.541206875Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.541390131Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.541408411Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.541472263Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.541517444Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.541608206Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.541642427Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.541686799Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54173927Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.541804152Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.541832563Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.541951266Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542038498Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54211822Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542159522Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542204743Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542258775Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542289325Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542375608Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542435859Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54246681Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542559053Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542624785Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542659715Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542766468Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.542859291Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.543045186Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.543115628Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54320001Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.543330784Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.543522219Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.543597111Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.543640793Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.543691064Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.543732885Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.543774846Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.543809907Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54391529Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.543990412Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544036043Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544077234Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544123866Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544166147Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544226918Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54427111Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544315291Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544356762Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544399493Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544407674Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544435614Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544580548Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544751923Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.544907917Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.545094822Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.545395951Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.545443032Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.545500793Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.545531565Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.545643717Z namespaces/openshift-dns/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.545699399Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54574509Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.545778221Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.546537642Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.546576323Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.546721657Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.546796059Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.546896762Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.546949333Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.546989734Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.547068357Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.547085457Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.547158819Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.547227041Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.547263782Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.547348924Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.547416256Z namespaces/openshift-dns/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.547464098Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54756585Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.547647633Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.547884959Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.547967011Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.548249959Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.548379953Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.548564418Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54862537Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.548666481Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.548735782Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.548775354Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.548858426Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.548902167Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.54899076Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549065772Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549134304Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549168134Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549227486Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549268327Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549304858Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549351119Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549401101Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549410221Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549431272Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549595066Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549770931Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.549926695Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550133721Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550342567Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550364467Z namespaces/openshift-dns/pods/dns-default-c4tmv/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550405508Z namespaces/openshift-dns/pods/dns-default-c4tmv/dns-default-c4tmv.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550496751Z namespaces/openshift-dns/pods/dns-default-c4tmv/dns/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550514642Z namespaces/openshift-dns/pods/dns-default-c4tmv/dns/dns/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550529842Z namespaces/openshift-dns/pods/dns-default-c4tmv/dns/dns/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550555783Z namespaces/openshift-dns/pods/dns-default-c4tmv/dns/dns/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550649845Z namespaces/openshift-dns/pods/dns-default-c4tmv/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550721957Z namespaces/openshift-dns/pods/dns-default-c4tmv/dns/dns/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550766238Z namespaces/openshift-dns/pods/dns-default-c4tmv/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550775159Z namespaces/openshift-dns/pods/dns-default-c4tmv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550787809Z namespaces/openshift-dns/pods/dns-default-c4tmv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55082364Z namespaces/openshift-dns/pods/dns-default-c4tmv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.550949173Z namespaces/openshift-dns/pods/dns-default-c4tmv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551015105Z namespaces/openshift-dns/pods/dns-default-c4tmv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551030266Z namespaces/openshift-dns/pods/dns-default-n2l79/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551080797Z namespaces/openshift-dns/pods/dns-default-n2l79/dns-default-n2l79.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551166519Z namespaces/openshift-dns/pods/dns-default-n2l79/dns/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55117521Z namespaces/openshift-dns/pods/dns-default-n2l79/dns/dns/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55119035Z namespaces/openshift-dns/pods/dns-default-n2l79/dns/dns/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55120797Z namespaces/openshift-dns/pods/dns-default-n2l79/dns/dns/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551324434Z namespaces/openshift-dns/pods/dns-default-n2l79/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551397066Z namespaces/openshift-dns/pods/dns-default-n2l79/dns/dns/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551433987Z namespaces/openshift-dns/pods/dns-default-n2l79/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551442757Z namespaces/openshift-dns/pods/dns-default-n2l79/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551450337Z namespaces/openshift-dns/pods/dns-default-n2l79/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551495389Z namespaces/openshift-dns/pods/dns-default-n2l79/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551599341Z namespaces/openshift-dns/pods/dns-default-n2l79/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551665503Z namespaces/openshift-dns/pods/dns-default-n2l79/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551714644Z namespaces/openshift-dns/pods/dns-default-tgn5c/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551749845Z namespaces/openshift-dns/pods/dns-default-tgn5c/dns-default-tgn5c.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551828598Z namespaces/openshift-dns/pods/dns-default-tgn5c/dns/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551850798Z namespaces/openshift-dns/pods/dns-default-tgn5c/dns/dns/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.551872729Z namespaces/openshift-dns/pods/dns-default-tgn5c/dns/dns/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552026703Z namespaces/openshift-dns/pods/dns-default-tgn5c/dns/dns/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552126716Z namespaces/openshift-dns/pods/dns-default-tgn5c/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552183887Z namespaces/openshift-dns/pods/dns-default-tgn5c/dns/dns/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552233999Z namespaces/openshift-dns/pods/dns-default-tgn5c/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552247509Z namespaces/openshift-dns/pods/dns-default-tgn5c/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552256379Z namespaces/openshift-dns/pods/dns-default-tgn5c/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55228452Z namespaces/openshift-dns/pods/dns-default-tgn5c/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552382133Z namespaces/openshift-dns/pods/dns-default-tgn5c/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552464575Z namespaces/openshift-dns/pods/dns-default-tgn5c/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552480705Z namespaces/openshift-dns/pods/node-resolver-2wm8s/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552529397Z namespaces/openshift-dns/pods/node-resolver-2wm8s/node-resolver-2wm8s.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.5526259Z namespaces/openshift-dns/pods/node-resolver-2wm8s/dns-node-resolver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.5526366Z namespaces/openshift-dns/pods/node-resolver-2wm8s/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55264362Z namespaces/openshift-dns/pods/node-resolver-2wm8s/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552671471Z namespaces/openshift-dns/pods/node-resolver-2wm8s/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552770913Z namespaces/openshift-dns/pods/node-resolver-2wm8s/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552850756Z namespaces/openshift-dns/pods/node-resolver-2wm8s/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552911557Z namespaces/openshift-dns/pods/node-resolver-lq478/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.552972639Z namespaces/openshift-dns/pods/node-resolver-lq478/node-resolver-lq478.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553052071Z namespaces/openshift-dns/pods/node-resolver-lq478/dns-node-resolver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553063881Z namespaces/openshift-dns/pods/node-resolver-lq478/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553069012Z namespaces/openshift-dns/pods/node-resolver-lq478/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553106613Z namespaces/openshift-dns/pods/node-resolver-lq478/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553189205Z namespaces/openshift-dns/pods/node-resolver-lq478/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553254567Z namespaces/openshift-dns/pods/node-resolver-lq478/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553292818Z namespaces/openshift-dns/pods/node-resolver-p9gnp/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553333299Z namespaces/openshift-dns/pods/node-resolver-p9gnp/node-resolver-p9gnp.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553422462Z namespaces/openshift-dns/pods/node-resolver-p9gnp/dns-node-resolver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553430312Z namespaces/openshift-dns/pods/node-resolver-p9gnp/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553439342Z namespaces/openshift-dns/pods/node-resolver-p9gnp/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553455552Z namespaces/openshift-dns/pods/node-resolver-p9gnp/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553558835Z namespaces/openshift-dns/pods/node-resolver-p9gnp/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553627827Z namespaces/openshift-dns/pods/node-resolver-p9gnp/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553676748Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553718959Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.5537535Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553801772Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553912835Z namespaces/openshift-etcd/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.553966666Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554020438Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554074919Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554131161Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554175112Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554260334Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554337956Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554418119Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55446639Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554508111Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554552013Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554593504Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554668266Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554713387Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554766098Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554861551Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554947083Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.554990374Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.555090557Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55516891Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.555249252Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.555327994Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.555408076Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.555527119Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.555688754Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.555749726Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.555799847Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.555856528Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55591642Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.555954061Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556000212Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556078655Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556160577Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556222148Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55626909Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556311291Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556353532Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556394503Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556403643Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556460275Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556605749Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556784964Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.556948208Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.557249677Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.557439942Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.557484203Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.557545725Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.557587376Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.557682139Z namespaces/openshift-host-network/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.557767021Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.557828432Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.557895785Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.557980017Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.558000117Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.558114391Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.558186573Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.558273025Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.558337577Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.558384098Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.5584463Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55846265Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.558569913Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.558622484Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.558672356Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.558744418Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55882299Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.558894072Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559008365Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559083397Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559167619Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559249762Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559325134Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559474168Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55956648Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559622002Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559665113Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559726685Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559764666Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559827537Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.55990506Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.559980411Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.560023063Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.560114725Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.560195257Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.560238569Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.56028524Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.560327981Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.560370012Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.560428594Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.560447695Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.560518826Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.560686211Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.560900537Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561053091Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561249206Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561436582Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561465852Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561524684Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561564255Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561653648Z namespaces/openshift-image-registry/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561696269Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561762091Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561804452Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561854193Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.561913845Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562039058Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562168952Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562355337Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562394788Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562424609Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.56247874Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562515741Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562623624Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562686286Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562720107Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562813389Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562878491Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562890772Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.562935533Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.563029455Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.563060486Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.564054544Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.564168527Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.564396793Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.564483686Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.564686641Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.564853785Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565062741Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565125653Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565170044Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565222896Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565271427Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565310648Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.56536851Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565452572Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565531284Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565594856Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565635847Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565721209Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.56576181Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565902194Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.565999917Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.566076889Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.56611827Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.566182432Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.566212263Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.566267595Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.566278585Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.566340037Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.566541022Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.566723757Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.566908362Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.567124338Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.567316743Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.567326544Z namespaces/openshift-image-registry/pods/image-registry-55ff9774c5-tss26/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.567337494Z namespaces/openshift-image-registry/pods/image-registry-55ff9774c5-tss26/image-registry-55ff9774c5-tss26.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.567464727Z namespaces/openshift-image-registry/pods/image-registry-55ff9774c5-tss26/registry/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.567475318Z namespaces/openshift-image-registry/pods/image-registry-55ff9774c5-tss26/registry/registry/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.567480568Z namespaces/openshift-image-registry/pods/image-registry-55ff9774c5-tss26/registry/registry/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.567523239Z namespaces/openshift-image-registry/pods/image-registry-55ff9774c5-tss26/registry/registry/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.5697394Z namespaces/openshift-image-registry/pods/image-registry-55ff9774c5-tss26/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.569813442Z namespaces/openshift-image-registry/pods/image-registry-55ff9774c5-tss26/registry/registry/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.569852613Z namespaces/openshift-image-registry/pods/node-ca-c4kct/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.569906615Z namespaces/openshift-image-registry/pods/node-ca-c4kct/node-ca-c4kct.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.569983087Z namespaces/openshift-image-registry/pods/node-ca-c4kct/node-ca/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.569993717Z namespaces/openshift-image-registry/pods/node-ca-c4kct/node-ca/node-ca/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570000167Z namespaces/openshift-image-registry/pods/node-ca-c4kct/node-ca/node-ca/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570011438Z namespaces/openshift-image-registry/pods/node-ca-c4kct/node-ca/node-ca/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570217273Z namespaces/openshift-image-registry/pods/node-ca-c4kct/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570285625Z namespaces/openshift-image-registry/pods/node-ca-c4kct/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570320306Z namespaces/openshift-image-registry/pods/node-ca-qs2dt/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570351867Z namespaces/openshift-image-registry/pods/node-ca-qs2dt/node-ca-qs2dt.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570433049Z namespaces/openshift-image-registry/pods/node-ca-qs2dt/node-ca/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570442359Z namespaces/openshift-image-registry/pods/node-ca-qs2dt/node-ca/node-ca/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57045407Z namespaces/openshift-image-registry/pods/node-ca-qs2dt/node-ca/node-ca/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570510711Z namespaces/openshift-image-registry/pods/node-ca-qs2dt/node-ca/node-ca/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570698476Z namespaces/openshift-image-registry/pods/node-ca-qs2dt/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570748588Z namespaces/openshift-image-registry/pods/node-ca-qs2dt/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570766578Z namespaces/openshift-image-registry/pods/node-ca-x8w62/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57082457Z namespaces/openshift-image-registry/pods/node-ca-x8w62/node-ca-x8w62.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570944723Z namespaces/openshift-image-registry/pods/node-ca-x8w62/node-ca/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570966334Z namespaces/openshift-image-registry/pods/node-ca-x8w62/node-ca/node-ca/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.570978564Z namespaces/openshift-image-registry/pods/node-ca-x8w62/node-ca/node-ca/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571000175Z namespaces/openshift-image-registry/pods/node-ca-x8w62/node-ca/node-ca/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571174299Z namespaces/openshift-image-registry/pods/node-ca-x8w62/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571240971Z namespaces/openshift-image-registry/pods/node-ca-x8w62/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571281722Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571312693Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571374395Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571411926Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571459367Z namespaces/openshift-infra/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571467478Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571472178Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571511449Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571661143Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571853138Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.571999652Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.572227348Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.572442064Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.572462425Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.572544667Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.572587518Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57265415Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.572696021Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.572803624Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.572902487Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.572984219Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573041181Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573094362Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573158344Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573207735Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573286848Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573337599Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57336649Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573458112Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573538134Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573583026Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573694159Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573779211Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.573946016Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.574030788Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.574197022Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.574317766Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.574500471Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.574554912Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.574655975Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.574708747Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.574759708Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57482173Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.574882441Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.574970804Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.575051346Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.575100747Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.575148969Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57518797Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.575236351Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.575292253Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.575308713Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.575365435Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57556925Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.575749325Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.575957471Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.576165757Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.576576188Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.576586138Z namespaces/openshift-ingress-canary/pods/ingress-canary-6ln68/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.576626479Z namespaces/openshift-ingress-canary/pods/ingress-canary-6ln68/ingress-canary-6ln68.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.576715702Z namespaces/openshift-ingress-canary/pods/ingress-canary-6ln68/serve-healthcheck-canary/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.576726412Z namespaces/openshift-ingress-canary/pods/ingress-canary-6ln68/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.576735952Z namespaces/openshift-ingress-canary/pods/ingress-canary-6ln68/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.576773873Z namespaces/openshift-ingress-canary/pods/ingress-canary-6ln68/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.576898537Z namespaces/openshift-ingress-canary/pods/ingress-canary-6ln68/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.576972689Z namespaces/openshift-ingress-canary/pods/ingress-canary-6ln68/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57702647Z namespaces/openshift-ingress-canary/pods/ingress-canary-h8xj8/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577059101Z namespaces/openshift-ingress-canary/pods/ingress-canary-h8xj8/ingress-canary-h8xj8.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577152194Z namespaces/openshift-ingress-canary/pods/ingress-canary-h8xj8/serve-healthcheck-canary/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577160624Z namespaces/openshift-ingress-canary/pods/ingress-canary-h8xj8/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577167724Z namespaces/openshift-ingress-canary/pods/ingress-canary-h8xj8/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577210725Z namespaces/openshift-ingress-canary/pods/ingress-canary-h8xj8/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577314358Z namespaces/openshift-ingress-canary/pods/ingress-canary-h8xj8/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57738613Z namespaces/openshift-ingress-canary/pods/ingress-canary-h8xj8/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577424531Z namespaces/openshift-ingress-canary/pods/ingress-canary-wjwjl/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577475603Z namespaces/openshift-ingress-canary/pods/ingress-canary-wjwjl/ingress-canary-wjwjl.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577551555Z namespaces/openshift-ingress-canary/pods/ingress-canary-wjwjl/serve-healthcheck-canary/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577560805Z namespaces/openshift-ingress-canary/pods/ingress-canary-wjwjl/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577572905Z namespaces/openshift-ingress-canary/pods/ingress-canary-wjwjl/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577623157Z namespaces/openshift-ingress-canary/pods/ingress-canary-wjwjl/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577715749Z namespaces/openshift-ingress-canary/pods/ingress-canary-wjwjl/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577787951Z namespaces/openshift-ingress-canary/pods/ingress-canary-wjwjl/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577823902Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577883544Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577951366Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.577972176Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57811092Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578162462Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578230784Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578271454Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578339876Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578364307Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57845179Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578534832Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578615174Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578674086Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578724187Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578781808Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.57882897Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578929933Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.578967364Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.579017895Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.579102707Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.579262162Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.579303483Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.580171907Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.580237309Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.580377663Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.580453465Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.580538137Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.58066842Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.580893497Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.580945048Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.580982419Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581054811Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581093412Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581156074Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581164054Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581212255Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581283167Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581322109Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581407181Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581486663Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581564755Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581583176Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581665998Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581687239Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581789861Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581804942Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581916895Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.581936665Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.582014597Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.582035428Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.582053209Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.582154821Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.582172582Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.582212353Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.582371277Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.582548832Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.582714997Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.582947383Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.583179169Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.583224971Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.583306823Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.583318493Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.583431016Z namespaces/openshift-ingress/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.583475848Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.583671303Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.583719244Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.583787736Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.583818507Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.583974401Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.584267189Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.584498626Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.584553297Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.584595228Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.58466032Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.584693631Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.584783064Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.584877806Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.584896896Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.58500149Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.585040771Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.585055041Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.585098992Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.585197145Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.585281407Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.58538197Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.585407481Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.585824482Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.585955556Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.586224903Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.586300465Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.586541132Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.586673816Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.586915412Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587008595Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587052736Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587143148Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587154599Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587164499Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587256512Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587275142Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587347394Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587394055Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587479418Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.58755686Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587606551Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587655163Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587713874Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587765075Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587828467Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587892759Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.5879361Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.587985532Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.588037513Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.588047413Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.588097524Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.588268959Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.588458694Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.588594458Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.588793184Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.58901519Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.58902562Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.589063231Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/istiod-openshift-gateway-5677ccb49b-zhst7.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.589190555Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.589204945Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.589210745Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.589244476Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.592691251Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.592762813Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-zhst7/discovery/discovery/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.592821474Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.592886266Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.593221785Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.593229336Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.593236076Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.593286857Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.593617806Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-8699b59bbc-s8kvl/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.593713659Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.59376151Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router-default-657b5f585b-96g68.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.593894584Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.593906014Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.593921085Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.593956196Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594084939Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594150811Z namespaces/openshift-ingress/pods/router-default-657b5f585b-96g68/router/router/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594203312Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594246594Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594308875Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594372937Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594456439Z namespaces/openshift-insights/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594509731Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594573553Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594605274Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594682716Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594713506Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594862241Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.594988704Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595101477Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595120358Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595175469Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595266692Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595311253Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595378025Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595413606Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595502258Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.59554712Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595591551Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595678533Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595758555Z namespaces/openshift-insights/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.595790836Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.596752473Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.596855005Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.597139823Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.597267077Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.597565805Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.597650677Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.597797721Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.597857303Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.597919585Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.597993167Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598039548Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598137321Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598197332Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598231123Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598292625Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598326106Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598403188Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598424719Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598513221Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598549492Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598643074Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598720387Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598780308Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598821719Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598909622Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.598947443Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.599029475Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.599074566Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.599137438Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.599158159Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.5992163Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.599229001Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.599285782Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.599478448Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.599682963Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.599998172Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.600207377Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.600443394Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.600456375Z namespaces/openshift-insights/pods/insights-operator-6f9c644dd9-ws6nf/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.600514126Z namespaces/openshift-insights/pods/insights-operator-6f9c644dd9-ws6nf/insights-operator-6f9c644dd9-ws6nf.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.600607108Z namespaces/openshift-insights/pods/insights-operator-6f9c644dd9-ws6nf/insights-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.600619279Z namespaces/openshift-insights/pods/insights-operator-6f9c644dd9-ws6nf/insights-operator/insights-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.600623989Z namespaces/openshift-insights/pods/insights-operator-6f9c644dd9-ws6nf/insights-operator/insights-operator/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.60066882Z namespaces/openshift-insights/pods/insights-operator-6f9c644dd9-ws6nf/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.601075832Z namespaces/openshift-insights/pods/insights-operator-6f9c644dd9-ws6nf/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.601369569Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.60139198Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/insights-runtime-extractor-6jk4g.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.601520064Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.601530334Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/exporter/exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.601537714Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/exporter/exporter/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.601545404Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/exporter/exporter/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.601653567Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.601719769Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/exporter/exporter/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.60173766Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/extractor/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6017421Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/extractor/extractor/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.60174816Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/extractor/extractor/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.601817102Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/extractor/extractor/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.601920715Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.601993567Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/extractor/extractor/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602054788Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602066509Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602071179Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.60211608Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602211893Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602284775Z namespaces/openshift-insights/pods/insights-runtime-extractor-6jk4g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602340336Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602386377Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/insights-runtime-extractor-tv5mn.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6024843Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602496391Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/exporter/exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602501291Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/exporter/exporter/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602536772Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/exporter/exporter/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602630824Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602697296Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/exporter/exporter/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602772838Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/extractor/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602781808Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/extractor/extractor/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602802389Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/extractor/extractor/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602812999Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/extractor/extractor/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602919082Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.602987524Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/extractor/extractor/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603025655Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603079887Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603090257Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603120868Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.60322194Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603290002Z namespaces/openshift-insights/pods/insights-runtime-extractor-tv5mn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603335844Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603370295Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/insights-runtime-extractor-xbczd.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603463187Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603471747Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/exporter/exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603476187Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/exporter/exporter/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603512919Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/exporter/exporter/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603608021Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603678183Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/exporter/exporter/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603691823Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/extractor/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603700224Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/extractor/extractor/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603707734Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/extractor/extractor/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603765846Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/extractor/extractor/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603854908Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.60394294Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/extractor/extractor/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603985412Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603993532Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.603997902Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604054483Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604125625Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604197927Z namespaces/openshift-insights/pods/insights-runtime-extractor-xbczd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604240568Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604265689Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604325721Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604363642Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604416843Z namespaces/openshift-keda/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604424073Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604428794Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604465515Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604556757Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604611629Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604618609Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.60464904Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604713081Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604793204Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604907307Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.604998029Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.605048701Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.605058821Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.605104682Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.605249896Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.605431831Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.605626867Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.605821362Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.605984616Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606270814Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606295805Z namespaces/openshift-keda/operators.coreos.com/installplans/install-gxjgw.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606431729Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606451189Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606515101Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606543082Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606606713Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606647865Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606741687Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606761168Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606866441Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606888431Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606968394Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.606985844Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607092177Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607176099Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607244321Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607298292Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607342504Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607386555Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607434526Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607516978Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6075667Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607609761Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607689413Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607748805Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.607795756Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6079226Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608008742Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608085874Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608166866Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608246218Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608370192Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608534717Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608602248Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608623459Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608700291Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608734322Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608790264Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608831874Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.608975609Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.609051711Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.609114962Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.609137193Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.609203965Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.609237376Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.609283097Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.609291007Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.609342629Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.609486863Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.609659017Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.609794431Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610008347Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610210903Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610252314Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610290815Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610339646Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610400628Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610443599Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610504511Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610547572Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610603273Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610644215Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610727007Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610807919Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610910522Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.610960923Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611006954Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611049226Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611092507Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611168739Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.61121303Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611263581Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611346314Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611415446Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611430136Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611549399Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611631051Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611704793Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611787146Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.611898799Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612016972Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612157826Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612219148Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612265359Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612319541Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612367022Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612423523Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612470505Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612547937Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612625849Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612689711Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612737432Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612791974Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612850945Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.612963208Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.613007389Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.613055051Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.613099442Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.613148613Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.613194174Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.613240176Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.613275077Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.613430651Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.613614236Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.61374591Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.613963736Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614166071Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614211093Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614257174Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614311845Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614408138Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614456559Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614525271Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614571312Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614630434Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614676355Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614760528Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.61485349Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.614936542Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615003994Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615045185Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615093967Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615131728Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.61521509Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615266211Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615312763Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615391475Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615449407Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615496708Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615602581Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615681553Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615762345Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.615858378Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616029832Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616148086Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.61631828Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616386382Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616407143Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616470935Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616516206Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616570197Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616604978Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616700911Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616778213Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616854265Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616910727Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.616970188Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.617014099Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.617065881Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.617114032Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.617153263Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.617164654Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.617203695Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.617355609Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.617532984Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.617665828Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.617879083Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618077529Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.61812076Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618184512Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618204392Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618291585Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618337596Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618437809Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.61847242Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618538301Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618570572Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618659635Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618738717Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618817889Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618896701Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618920372Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.618981994Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.619029715Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.619122428Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.619142658Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.61920055Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.619281692Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.619319183Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.619329323Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.619374604Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.619441726Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.619485998Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.61958299Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.619658572Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.620154346Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.620232298Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62031722Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.620434884Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.620592688Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.620639699Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62068308Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.620724822Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.620772423Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.620817384Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.620891946Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.620972828Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621056351Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621099732Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621145933Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621189414Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621231155Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621273787Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621281657Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621321288Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621466542Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621641827Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621777021Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.621994536Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622199652Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622224833Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622279104Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622296965Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622381337Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622416938Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62246564Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622516571Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622582343Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622602593Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622700816Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622778688Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62286066Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622924882Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.622979224Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623008074Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623063486Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623143908Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62319326Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623233791Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623329503Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623382915Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623417586Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623523458Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623601601Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623685253Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623763365Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.623866428Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624059793Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624221598Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624272929Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624321431Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624373982Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624412783Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624470535Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624523136Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624598458Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62467691Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624733052Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624752102Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624813874Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624905856Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624956838Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.624965588Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62501907Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.625164504Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62540869Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.625604086Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.625796831Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626005067Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626064128Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62611119Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626142501Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626214543Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626260544Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626307105Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626355427Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626409108Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626444959Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626531581Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626605443Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626697256Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626720717Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626773818Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62685622Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626890691Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.626985914Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.627033705Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.627076246Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.627167009Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62721659Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.627255091Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.627359484Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.627435266Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.627516978Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.627610371Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.627682503Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.627802506Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.627986441Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628029853Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628079204Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628134065Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628169316Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628222178Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628253499Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628344081Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628429724Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628470495Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628516936Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628564317Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628608778Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62865095Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62865989Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628711991Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.628860395Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.629054381Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.629186514Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.62937755Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.629559175Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.629591745Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.629651657Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.629683328Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6297617Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.629800061Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.629922554Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.629942045Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630038858Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630053398Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630156471Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630265774Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630372587Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630418958Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630448389Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630507001Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630544172Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630637284Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630684846Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630734637Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630814239Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630888071Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630902452Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.630943853Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.631011335Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.631040505Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.631156108Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.631240531Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.631408415Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.631487008Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.631627061Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.631780225Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.631993351Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632158316Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632195457Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632259969Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.63230131Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632336421Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632397773Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632484665Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632570517Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632625009Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632645939Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632724021Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632748292Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632808214Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632821994Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.632862345Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.633122812Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.633302497Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.633435751Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.633632706Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.633809611Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.633818412Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-55889f7b47-vl67p/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.633896384Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-55889f7b47-vl67p/kube-storage-version-migrator-operator-55889f7b47-vl67p.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634032697Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-55889f7b47-vl67p/kube-storage-version-migrator-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634041618Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-55889f7b47-vl67p/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634051248Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-55889f7b47-vl67p/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634095809Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-55889f7b47-vl67p/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634221592Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-55889f7b47-vl67p/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634335896Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634370457Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634416438Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634464189Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634530371Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634543372Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634609473Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634656615Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634723496Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634760067Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.63486613Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.634989984Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.635085686Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.635130377Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.635163778Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.63522247Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.635272561Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.635352034Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.635410045Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.635488387Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.63557535Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.635623651Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.635677833Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.635774445Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.635905849Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.636024872Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.636102314Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.636222678Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.636348251Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.636509775Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.636561877Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.636592978Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.63665679Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.636703401Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.636766073Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.636786203Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.636931427Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.637014889Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.637053441Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.637096012Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.637137943Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.637184644Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.637225415Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.637232975Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.637289857Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.637437051Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.637617456Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.63774526Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.637972356Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638269864Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638277864Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638318725Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/migrator-76794cbd6c-kq9nw.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638402538Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/graceful-termination/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638410118Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/graceful-termination/graceful-termination/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638422828Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638471129Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638553492Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638623564Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638656254Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/migrator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638664125Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/migrator/migrator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638672705Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/migrator/migrator/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638702706Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/migrator/migrator/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638800798Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638909821Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-76794cbd6c-kq9nw/migrator/migrator/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638929712Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.638971423Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639028955Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639067756Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639133808Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639170659Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639242761Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639290312Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639371834Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639410165Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639503538Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639633131Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639754335Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639802516Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639860138Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.63994375Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.639976221Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.640068173Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.640099894Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.640150876Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.640231858Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.640282639Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.640293139Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64031521Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.640413283Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.640467894Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.640520336Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.640626799Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.640720011Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64104808Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.641126333Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.641303897Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.641425431Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.641615216Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.641676888Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.641727209Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.641791611Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.641853933Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.641921114Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.641953225Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642051458Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6421252Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642176231Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642219772Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642273344Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642318315Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642369897Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642415168Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642459049Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642466069Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6425029Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642656584Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642827789Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.642998794Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64321724Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.643406885Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.643675823Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.643719364Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-mngj6.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.643792296Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.643834027Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.643922219Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64396158Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644017832Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644063313Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644131055Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644140865Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-crjpm/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644174096Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-crjpm/lws-controller-manager-7fdf8f455-crjpm.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644272089Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-crjpm/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644281729Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-crjpm/manager/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644286149Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-crjpm/manager/manager/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64432939Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-crjpm/manager/manager/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644500365Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-crjpm/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644570757Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-crjpm/manager/manager/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644615538Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-wdj86/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644651339Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-wdj86/lws-controller-manager-7fdf8f455-wdj86.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644738412Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-wdj86/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644749722Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-wdj86/manager/manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644754572Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-wdj86/manager/manager/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644793113Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-wdj86/manager/manager/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644912457Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-wdj86/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.644988689Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7fdf8f455-wdj86/manager/manager/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64502066Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-hrsdw/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645068461Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-hrsdw/openshift-lws-operator-fd8ccff4c-hrsdw.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645145903Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-hrsdw/openshift-lws-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645154563Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-hrsdw/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645169904Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-hrsdw/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645214255Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-hrsdw/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64538839Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-hrsdw/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645455552Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-hrsdw/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645474022Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645525183Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645564544Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645610006Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645644507Z namespaces/openshift-machine-api/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645652687Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645657877Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645697428Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.645867693Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.646094009Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.646232673Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.646426738Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.646607393Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.646617864Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.646623214Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.646631214Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.646778158Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64685228Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64687172Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.646929872Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.647077266Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.647250541Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.647385825Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64757961Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.647859058Z namespaces/openshift-marketplace/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.647873908Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.647889508Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64795557Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.648021822Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.648030302Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.648055833Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.648161806Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.648244498Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.64832668Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.648386142Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.648432203Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.648589858Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.648757252Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.648915527Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.649110172Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.649349159Z namespaces/openshift-monitoring/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.649385309Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.649459062Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.649508223Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.649564954Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.649614336Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.649747659Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.650185031Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.650526931Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.650745877Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.650785438Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.65085933Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.650908982Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.651019674Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.651080926Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.651129848Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.65120716Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.651255061Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.651301352Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.654269444Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.654413828Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.655434806Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.655507648Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.656465004Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.656579197Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.65705869Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.657223295Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.657268706Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.65741267Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.657445451Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.657496552Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.657544564Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.657628796Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.657706078Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.657748799Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.657830652Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.658079979Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.658102339Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.658201212Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.658240343Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.658353216Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.658390877Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.65849521Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.658656224Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.658746037Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.659026285Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.659162418Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.659298082Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.659385874Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.659493477Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.65958323Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.659659262Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.659695473Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.659795456Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.659902049Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.659985571Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.660093254Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.660171926Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66032133Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.660412973Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.660505625Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.660591828Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66068046Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.660773102Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.660854255Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.660895456Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.660949737Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.660960938Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.661008679Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.661211805Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66142288Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.661568494Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6617703Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.661999706Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662043488Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66215332Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662161971Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662226033Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662396107Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662404747Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662408747Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662446948Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662545821Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662612003Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662645174Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662653524Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662657444Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662692315Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662796718Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66288194Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662930032Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662941182Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662946082Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.662983203Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663067006Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663138058Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663184089Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663193169Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663197069Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66322879Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663322073Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663390465Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663422495Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663429986Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663439086Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663482987Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6635768Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663639121Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663675242Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663683353Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663687273Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663728334Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663819936Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66396816Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663988341Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.663999461Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.664008282Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.664060703Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.664154386Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.664222697Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.664271609Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-847b8bd8d7-pmrll/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66430846Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-847b8bd8d7-pmrll/cluster-monitoring-operator-847b8bd8d7-pmrll.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.664419093Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-847b8bd8d7-pmrll/cluster-monitoring-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.664427583Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-847b8bd8d7-pmrll/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.664431803Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-847b8bd8d7-pmrll/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.664476604Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-847b8bd8d7-pmrll/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665256236Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-847b8bd8d7-pmrll/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665328038Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-847b8bd8d7-pmrll/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665375309Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66542192Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-state-metrics-84c885658c-cb4bk.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665516063Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-main/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665531463Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665539474Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665585965Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665676427Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665747239Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66578406Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-self/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6657924Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665796341Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665825952Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.665943955Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666009826Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666057968Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-state-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666066438Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666072838Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666113499Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666312805Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666380067Z namespaces/openshift-monitoring/pods/kube-state-metrics-84c885658c-cb4bk/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666421658Z namespaces/openshift-monitoring/pods/metrics-server-7cc9c9ffcf-xttz5/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666452489Z namespaces/openshift-monitoring/pods/metrics-server-7cc9c9ffcf-xttz5/metrics-server-7cc9c9ffcf-xttz5.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666549501Z namespaces/openshift-monitoring/pods/metrics-server-7cc9c9ffcf-xttz5/metrics-server/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666558042Z namespaces/openshift-monitoring/pods/metrics-server-7cc9c9ffcf-xttz5/metrics-server/metrics-server/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666563952Z namespaces/openshift-monitoring/pods/metrics-server-7cc9c9ffcf-xttz5/metrics-server/metrics-server/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666595993Z namespaces/openshift-monitoring/pods/metrics-server-7cc9c9ffcf-xttz5/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666691035Z namespaces/openshift-monitoring/pods/metrics-server-7cc9c9ffcf-xttz5/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666765167Z namespaces/openshift-monitoring/pods/metrics-server-7cc9c9ffcf-xttz5/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666807508Z namespaces/openshift-monitoring/pods/monitoring-plugin-5984459ff8-492lt/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66686059Z namespaces/openshift-monitoring/pods/monitoring-plugin-5984459ff8-492lt/monitoring-plugin-5984459ff8-492lt.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666954643Z namespaces/openshift-monitoring/pods/monitoring-plugin-5984459ff8-492lt/monitoring-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666970993Z namespaces/openshift-monitoring/pods/monitoring-plugin-5984459ff8-492lt/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.666977063Z namespaces/openshift-monitoring/pods/monitoring-plugin-5984459ff8-492lt/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667068216Z namespaces/openshift-monitoring/pods/monitoring-plugin-5984459ff8-492lt/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667172338Z namespaces/openshift-monitoring/pods/monitoring-plugin-5984459ff8-492lt/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667246131Z namespaces/openshift-monitoring/pods/monitoring-plugin-5984459ff8-492lt/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667262391Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667313672Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/node-exporter-9p8rv.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667420955Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/init-textfile/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667430396Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/init-textfile/init-textfile/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667441276Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/init-textfile/init-textfile/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667488187Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667565019Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667640991Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667674302Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667681992Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667693983Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667739824Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667834597Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667921239Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66796597Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/node-exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66797394Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/node-exporter/node-exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.667982161Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/node-exporter/node-exporter/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668035462Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668143135Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668219167Z namespaces/openshift-monitoring/pods/node-exporter-9p8rv/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668267089Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66831171Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/node-exporter-nv98w.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668413093Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/init-textfile/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668422023Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/init-textfile/init-textfile/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668426303Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/init-textfile/init-textfile/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668457814Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668536466Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668604818Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668641999Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668650929Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668656899Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668709261Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668805193Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668890996Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668932627Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/node-exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668942337Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/node-exporter/node-exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668946607Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/node-exporter/node-exporter/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.668980988Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669089761Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669159963Z namespaces/openshift-monitoring/pods/node-exporter-nv98w/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669203784Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669244755Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/node-exporter-pfl9r.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669336028Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/init-textfile/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669362229Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/init-textfile/init-textfile/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669367709Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/init-textfile/init-textfile/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.66940106Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669471702Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669550514Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669582635Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669593805Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669599675Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669641926Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669734939Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669803881Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669868333Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/node-exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669883733Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/node-exporter/node-exporter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669889663Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/node-exporter/node-exporter/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.669918994Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670012016Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670099019Z namespaces/openshift-monitoring/pods/node-exporter-pfl9r/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67013983Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670187301Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/openshift-state-metrics-86c5bc79d6-qhv2n.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670321435Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-main/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670330665Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670334676Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670371757Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670471249Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670540741Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670572302Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-self/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670580512Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670587102Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670625833Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670722136Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670790848Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67085043Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/openshift-state-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67086365Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67086784Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.670908061Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671001664Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671077436Z namespaces/openshift-monitoring/pods/openshift-state-metrics-86c5bc79d6-qhv2n/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671131987Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671195989Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671371294Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671379414Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671383474Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671419095Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671520608Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67159082Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671630151Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671638551Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671645052Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671687393Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671779415Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671849917Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671903979Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671913309Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.671931559Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67196275Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672051323Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672118545Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672159186Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672167556Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672171396Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672205627Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672290539Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672358431Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672404872Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672413983Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672417973Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672451174Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672541586Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672610478Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672636889Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672642569Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.672646379Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67269062Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.673528123Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.673579765Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.673589325Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.673593245Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.673647136Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.673741719Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.673809431Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.673849622Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.673918344Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/prometheus-operator-7bd5d56d6-pk5s2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.673999256Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674007586Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674011466Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674050108Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67415036Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674219102Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674249063Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/prometheus-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674257233Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674263524Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674303735Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674571892Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674639174Z namespaces/openshift-monitoring/pods/prometheus-operator-7bd5d56d6-pk5s2/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674677275Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674727046Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67485968Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c/prometheus-operator-admission-webhook/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67487161Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674879321Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.674929662Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675026105Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675094706Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-5c6747dc7b-lqv8c/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675137997Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675174058Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/telemeter-client-5644b867f7-lxt8l.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675325173Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675335723Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675340643Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675372224Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675465746Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675540789Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675556659Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/reload/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675566289Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/reload/reload/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67559998Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/reload/reload/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675654752Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/reload/reload/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675741434Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675813586Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/reload/reload/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675830676Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/telemeter-client/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675855597Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/telemeter-client/telemeter-client/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675864428Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.675912329Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676082214Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676152245Z namespaces/openshift-monitoring/pods/telemeter-client-5644b867f7-lxt8l/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676179466Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676254678Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/thanos-querier-5c9d7f8c77-5p257.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676381352Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676389852Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676394012Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676434283Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676531376Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676599858Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67669124Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-rules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676699631Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676704111Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676740782Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676849024Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676927397Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676966918Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-web/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676975958Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.676980288Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677018339Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677104682Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677178184Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677210214Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677219115Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677223815Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677262946Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677351268Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67742422Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677455851Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/prom-label-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677464592Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677469372Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677504853Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677595405Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677664247Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677695798Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/thanos-query/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677704748Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/thanos-query/thanos-query/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677709508Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/thanos-query/thanos-query/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67775224Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677896283Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.677959425Z namespaces/openshift-monitoring/pods/thanos-querier-5c9d7f8c77-5p257/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678000126Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678055848Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678104999Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67815107Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678260533Z namespaces/openshift-multus/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678272814Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678351476Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678391687Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678438628Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67848046Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678674045Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678751167Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678830169Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678895791Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.678945492Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.679193159Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.679319803Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.679381704Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.679430606Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.679475427Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.679523278Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67960588Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.679655522Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.679701343Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.679777555Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.679856237Z namespaces/openshift-multus/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.67995492Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.680101704Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.680184636Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.680639239Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.680761122Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.681319617Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6814053Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.681540313Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.681590575Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.681633636Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.681728429Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.681811281Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.681919414Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682010626Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682076428Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682158001Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682251803Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682324125Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682378247Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682460829Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.68251149Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682557912Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682620103Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682654854Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682708385Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682757327Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682833429Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682931392Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.682983543Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.683029444Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.683078846Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.683122287Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.683174288Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.68321869Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.683264171Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.683278961Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.683321242Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.683471086Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.683733374Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.683903598Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.684098294Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.684285949Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.684295999Z namespaces/openshift-multus/pods/multus-2979h/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.684303709Z namespaces/openshift-multus/pods/multus-2979h/multus-2979h.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.684420023Z namespaces/openshift-multus/pods/multus-2979h/kube-multus/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.684430113Z namespaces/openshift-multus/pods/multus-2979h/kube-multus/kube-multus/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.684434983Z namespaces/openshift-multus/pods/multus-2979h/kube-multus/kube-multus/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.684467514Z namespaces/openshift-multus/pods/multus-2979h/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.68505767Z namespaces/openshift-multus/pods/multus-2979h/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.685126672Z namespaces/openshift-multus/pods/multus-2979h/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.685144203Z namespaces/openshift-multus/pods/multus-76p9s/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.685197374Z namespaces/openshift-multus/pods/multus-76p9s/multus-76p9s.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.685297527Z namespaces/openshift-multus/pods/multus-76p9s/kube-multus/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.685308297Z namespaces/openshift-multus/pods/multus-76p9s/kube-multus/kube-multus/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.685313517Z namespaces/openshift-multus/pods/multus-76p9s/kube-multus/kube-multus/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.685353758Z namespaces/openshift-multus/pods/multus-76p9s/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686278674Z namespaces/openshift-multus/pods/multus-76p9s/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686352396Z namespaces/openshift-multus/pods/multus-76p9s/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686400067Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686444118Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/multus-additional-cni-plugins-bgqzs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686592482Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/bond-cni-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686601993Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686609983Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686667134Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686762267Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6868571Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.68688291Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686892991Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/cni-plugins/cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686899971Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.686948312Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687034715Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687106366Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687151228Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/egress-router-binary-copy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687167098Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687171818Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687179859Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687278641Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687353123Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687383414Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687391154Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687395465Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687439866Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687514448Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6875845Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687628991Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/routeoverride-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687638501Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687642681Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687668522Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687761764Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.687825086Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.68795857Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni-bincopy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6879716Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6879757Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688005931Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688099334Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688167166Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688208957Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688217437Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688221667Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688255058Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688350361Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688417073Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-bgqzs/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688457254Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688502005Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/multus-additional-cni-plugins-fn6vl.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688625898Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/bond-cni-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688635679Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688640389Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.68866868Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688771462Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688865145Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688897246Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688902936Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/cni-plugins/cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688907216Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.688950417Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.68904286Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689112892Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689147183Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/egress-router-binary-copy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689155483Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689159403Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689212344Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689298107Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689352568Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6893966Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.68940894Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.68941389Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689442951Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689518243Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689588795Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689625216Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/routeoverride-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689633446Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689639826Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689688937Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.68978343Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689848332Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689922434Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni-bincopy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689938364Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.689950445Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690206712Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690218642Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690225652Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690229632Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690233412Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690237272Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690268773Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690345766Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690422678Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fn6vl/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690437558Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.69049834Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/multus-additional-cni-plugins-m2pkz.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690621143Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/bond-cni-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690629433Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690633283Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690667064Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690760797Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690824409Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.69086716Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.69088405Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/cni-plugins/cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690897341Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.690925391Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691013144Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691079576Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691115757Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/egress-router-binary-copy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691124007Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691127877Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691161298Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.69125134Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691320102Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691362414Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691370614Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691374574Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691388974Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691470917Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691544879Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691572589Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/routeoverride-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691579709Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.69158374Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691632241Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691708013Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691774595Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691816506Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni-bincopy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691824506Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691828456Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691886048Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.691987371Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692039422Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692084183Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692093604Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692097574Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692104374Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692208397Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692275979Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-m2pkz/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692303259Z namespaces/openshift-multus/pods/multus-slc8t/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692360421Z namespaces/openshift-multus/pods/multus-slc8t/multus-slc8t.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692434513Z namespaces/openshift-multus/pods/multus-slc8t/kube-multus/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692444453Z namespaces/openshift-multus/pods/multus-slc8t/kube-multus/kube-multus/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692450093Z namespaces/openshift-multus/pods/multus-slc8t/kube-multus/kube-multus/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.692490174Z namespaces/openshift-multus/pods/multus-slc8t/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693450881Z namespaces/openshift-multus/pods/multus-slc8t/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693520803Z namespaces/openshift-multus/pods/multus-slc8t/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693560944Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693592805Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/network-metrics-daemon-nwtqb.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693691778Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693701588Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693706628Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693746299Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693862232Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693935494Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693950115Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/network-metrics-daemon/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693974475Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.693983215Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694054688Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694248713Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694316325Z namespaces/openshift-multus/pods/network-metrics-daemon-nwtqb/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694359376Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694405587Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/network-metrics-daemon-vqfxz.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.69449483Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.69451056Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694517Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694530351Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694640314Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694707315Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694720946Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/network-metrics-daemon/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694729286Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694733406Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694786558Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.694959862Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695018874Z namespaces/openshift-multus/pods/network-metrics-daemon-vqfxz/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695048255Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695093706Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/network-metrics-daemon-wpmw9.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695175098Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695184219Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695188059Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.69522251Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695312262Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695387374Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695399425Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/network-metrics-daemon/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695421615Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695433795Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695481427Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695659542Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695768935Z namespaces/openshift-multus/pods/network-metrics-daemon-wpmw9/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695795765Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695858907Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.69597843Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.695987681Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696026432Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696118114Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696163466Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696200947Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696287489Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.6963256Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696374801Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696412632Z namespaces/openshift-must-gather-s6w2r/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696420563Z namespaces/openshift-must-gather-s6w2r/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696427173Z namespaces/openshift-must-gather-s6w2r/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696475804Z namespaces/openshift-must-gather-s6w2r/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696648089Z namespaces/openshift-must-gather-s6w2r/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.696855794Z namespaces/openshift-must-gather-s6w2r/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.697002689Z namespaces/openshift-must-gather-s6w2r/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.697207374Z namespaces/openshift-must-gather-s6w2r/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.69743539Z namespaces/openshift-network-console/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.697476022Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.697541004Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.697578145Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.697638396Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.697668457Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.697755009Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.697886953Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.697992376Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698039877Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698089519Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698164941Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698196722Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698234612Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698288834Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698365506Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698413007Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698449668Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698537521Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698572422Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698619163Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698739956Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698817659Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.698952652Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699032705Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699281411Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699407195Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699523078Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699577629Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699618281Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699677522Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699732634Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699777715Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699807106Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699889708Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699936899Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.699980271Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700020982Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700107914Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700186696Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700235847Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700276639Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70032477Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700378702Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700421943Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700431043Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700468774Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700626548Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700803563Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.700961338Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701158673Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701339688Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701347958Z namespaces/openshift-network-console/pods/networking-console-plugin-6446c8d75d-v598q/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701385159Z namespaces/openshift-network-console/pods/networking-console-plugin-6446c8d75d-v598q/networking-console-plugin-6446c8d75d-v598q.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701467491Z namespaces/openshift-network-console/pods/networking-console-plugin-6446c8d75d-v598q/networking-console-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701476562Z namespaces/openshift-network-console/pods/networking-console-plugin-6446c8d75d-v598q/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701480522Z namespaces/openshift-network-console/pods/networking-console-plugin-6446c8d75d-v598q/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701502272Z namespaces/openshift-network-console/pods/networking-console-plugin-6446c8d75d-v598q/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701607535Z namespaces/openshift-network-console/pods/networking-console-plugin-6446c8d75d-v598q/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701675717Z namespaces/openshift-network-console/pods/networking-console-plugin-6446c8d75d-v598q/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701719348Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701757549Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701805951Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701849312Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701921874Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.701963895Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702031957Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702109729Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70215163Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702200172Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702310684Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702411337Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70251183Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702543911Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702592622Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702661554Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702680105Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702771277Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702789558Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70285429Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702893951Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.702988793Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.703030014Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.703065145Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.703146487Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.703182188Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70323474Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.703346353Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.703431915Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.703676352Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.703750154Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704060092Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704144285Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704247908Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704304749Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70433974Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704396102Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704435873Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704519155Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704555956Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704607927Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704668969Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704718411Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704750242Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704801103Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704911116Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.704984808Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.705020189Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70507221Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.705126982Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.705177173Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.705236465Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.705254735Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.705326307Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.705338678Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.705348468Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.705503212Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.705802911Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.705962595Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70616585Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.706342255Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.706353795Z namespaces/openshift-network-diagnostics/pods/network-check-source-7cbc45fddc-2ddwm/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.706403447Z namespaces/openshift-network-diagnostics/pods/network-check-source-7cbc45fddc-2ddwm/network-check-source-7cbc45fddc-2ddwm.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.706484069Z namespaces/openshift-network-diagnostics/pods/network-check-source-7cbc45fddc-2ddwm/check-endpoints/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70649277Z namespaces/openshift-network-diagnostics/pods/network-check-source-7cbc45fddc-2ddwm/check-endpoints/check-endpoints/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70649667Z namespaces/openshift-network-diagnostics/pods/network-check-source-7cbc45fddc-2ddwm/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70652789Z namespaces/openshift-network-diagnostics/pods/network-check-source-7cbc45fddc-2ddwm/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.706648204Z namespaces/openshift-network-diagnostics/pods/network-check-source-7cbc45fddc-2ddwm/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.706720386Z namespaces/openshift-network-diagnostics/pods/network-check-source-7cbc45fddc-2ddwm/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.706737996Z namespaces/openshift-network-diagnostics/pods/network-check-target-4nh6z/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.706795258Z namespaces/openshift-network-diagnostics/pods/network-check-target-4nh6z/network-check-target-4nh6z.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70688538Z namespaces/openshift-network-diagnostics/pods/network-check-target-4nh6z/network-check-target-container/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70689845Z namespaces/openshift-network-diagnostics/pods/network-check-target-4nh6z/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.706902601Z namespaces/openshift-network-diagnostics/pods/network-check-target-4nh6z/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.706936192Z namespaces/openshift-network-diagnostics/pods/network-check-target-4nh6z/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707027894Z namespaces/openshift-network-diagnostics/pods/network-check-target-4nh6z/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707096446Z namespaces/openshift-network-diagnostics/pods/network-check-target-4nh6z/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707128007Z namespaces/openshift-network-diagnostics/pods/network-check-target-6bl8g/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707164618Z namespaces/openshift-network-diagnostics/pods/network-check-target-6bl8g/network-check-target-6bl8g.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.7072416Z namespaces/openshift-network-diagnostics/pods/network-check-target-6bl8g/network-check-target-container/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70724996Z namespaces/openshift-network-diagnostics/pods/network-check-target-6bl8g/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70725391Z namespaces/openshift-network-diagnostics/pods/network-check-target-6bl8g/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707284541Z namespaces/openshift-network-diagnostics/pods/network-check-target-6bl8g/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707378684Z namespaces/openshift-network-diagnostics/pods/network-check-target-6bl8g/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707451836Z namespaces/openshift-network-diagnostics/pods/network-check-target-6bl8g/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707487067Z namespaces/openshift-network-diagnostics/pods/network-check-target-x84ph/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707525818Z namespaces/openshift-network-diagnostics/pods/network-check-target-x84ph/network-check-target-x84ph.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70760474Z namespaces/openshift-network-diagnostics/pods/network-check-target-x84ph/network-check-target-container/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70761282Z namespaces/openshift-network-diagnostics/pods/network-check-target-x84ph/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70761663Z namespaces/openshift-network-diagnostics/pods/network-check-target-x84ph/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707653241Z namespaces/openshift-network-diagnostics/pods/network-check-target-x84ph/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707742214Z namespaces/openshift-network-diagnostics/pods/network-check-target-x84ph/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707808635Z namespaces/openshift-network-diagnostics/pods/network-check-target-x84ph/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707827386Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707898318Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707953119Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70796237Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.707998131Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708083503Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708131375Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708179386Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708259048Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708297149Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70834438Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708392432Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708440893Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708496354Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708544106Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708585837Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708632628Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708711471Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708793733Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708908116Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708931837Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.708982838Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709029279Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70907177Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709150823Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709198994Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709239255Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709324287Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709369669Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709378359Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70941739Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709467321Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709514613Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709605385Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709685947Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.70977518Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709855392Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.709955495Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710107549Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710219402Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710266243Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710313735Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710367106Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710412667Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710458088Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.71050028Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710547171Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710588502Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710669114Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710747696Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710793118Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710835059Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710898631Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710933952Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710988913Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.710998513Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.711030774Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.711179748Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.711358183Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.711501777Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.711688222Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.711950269Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.711996181Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712044342Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712055772Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712093464Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712148545Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712186056Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712236487Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712278938Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.71232661Z namespaces/openshift-network-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712371911Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712441983Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712457383Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712525695Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712560336Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712768902Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712861374Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712953007Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.712972958Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713020569Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713082101Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713135602Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713169323Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713214654Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713301937Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713347058Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713389609Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713470671Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713520453Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713529653Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713576094Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713626655Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713661027Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713798Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.713883943Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.714057767Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.71413606Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.714312674Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.714395037Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.71452597Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.714575852Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.714622673Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.714704045Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.714765247Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.714813308Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.71485966Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.714924231Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.714973733Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715012704Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715063075Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715103316Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715187769Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.71526256Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715318072Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715347803Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715402644Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715445956Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715518528Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715555609Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.71560961Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715650681Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715696112Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715708233Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715744844Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.715925769Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.716126064Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.716304369Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.716520975Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.716734661Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.716748072Z namespaces/openshift-network-operator/pods/iptables-alerter-7mx7p/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.716803843Z namespaces/openshift-network-operator/pods/iptables-alerter-7mx7p/iptables-alerter-7mx7p.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.716911636Z namespaces/openshift-network-operator/pods/iptables-alerter-7mx7p/iptables-alerter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.716924756Z namespaces/openshift-network-operator/pods/iptables-alerter-7mx7p/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.716943647Z namespaces/openshift-network-operator/pods/iptables-alerter-7mx7p/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.716982158Z namespaces/openshift-network-operator/pods/iptables-alerter-7mx7p/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717117711Z namespaces/openshift-network-operator/pods/iptables-alerter-7mx7p/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717203554Z namespaces/openshift-network-operator/pods/iptables-alerter-7mx7p/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717239385Z namespaces/openshift-network-operator/pods/iptables-alerter-cphgh/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717289866Z namespaces/openshift-network-operator/pods/iptables-alerter-cphgh/iptables-alerter-cphgh.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717364378Z namespaces/openshift-network-operator/pods/iptables-alerter-cphgh/iptables-alerter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717372789Z namespaces/openshift-network-operator/pods/iptables-alerter-cphgh/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717376799Z namespaces/openshift-network-operator/pods/iptables-alerter-cphgh/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.71741125Z namespaces/openshift-network-operator/pods/iptables-alerter-cphgh/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717505742Z namespaces/openshift-network-operator/pods/iptables-alerter-cphgh/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717573574Z namespaces/openshift-network-operator/pods/iptables-alerter-cphgh/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717613335Z namespaces/openshift-network-operator/pods/iptables-alerter-gzrln/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717647906Z namespaces/openshift-network-operator/pods/iptables-alerter-gzrln/iptables-alerter-gzrln.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717721978Z namespaces/openshift-network-operator/pods/iptables-alerter-gzrln/iptables-alerter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717729888Z namespaces/openshift-network-operator/pods/iptables-alerter-gzrln/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717733748Z namespaces/openshift-network-operator/pods/iptables-alerter-gzrln/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717761519Z namespaces/openshift-network-operator/pods/iptables-alerter-gzrln/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717853512Z namespaces/openshift-network-operator/pods/iptables-alerter-gzrln/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717939074Z namespaces/openshift-network-operator/pods/iptables-alerter-gzrln/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.717976325Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.718008096Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.718077978Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.718115589Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.7181621Z namespaces/openshift-node/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.718170011Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.718173991Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.718208741Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.718371896Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.718546901Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.718680034Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.719164748Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.719362363Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.719481447Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.719542068Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.71958971Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.719622391Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.719673582Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.719752454Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.719831566Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.719931719Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.719990401Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720009691Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720065263Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720091443Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720193006Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720209817Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720261218Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72034395Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720379481Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720437573Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720553056Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720635168Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720746281Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720819863Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.720936067Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72106544Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721233535Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721295746Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721344408Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721396079Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72142523Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721483502Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721540433Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721644486Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721723328Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721768979Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721810461Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721861792Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721922954Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.721981235Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.722038247Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.722100188Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.722110279Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72214418Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.722297464Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.722475979Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.722612073Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72286272Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723047755Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723096916Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723161148Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723181488Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.7232523Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723267971Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723355303Z namespaces/openshift-operators/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723371554Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723378754Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723396904Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723528698Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723711783Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.723879368Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.724211187Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.724394442Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.725168733Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.725211224Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.725278756Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.725321887Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.725384119Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72541842Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.725469321Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.725648066Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.725734199Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.725812391Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.725867352Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.725935224Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726074498Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726107909Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72614504Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726208212Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726279794Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726330855Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726351445Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726447578Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726488659Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726498649Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72651481Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726589322Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726605643Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726809098Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.726918361Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72725839Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.727385694Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.727859397Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728037442Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728224047Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728289199Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72833233Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728425082Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728537945Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728613747Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72870888Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728742961Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728776072Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728862734Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728890185Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728962437Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.728984788Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.7290639Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72907389Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.72908191Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729148732Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729266576Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729524892Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729565394Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729619755Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729656196Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729720458Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729757969Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729861652Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729910643Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729962695Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729973465Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.729989705Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.730108009Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73015469Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.730187591Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.730224222Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.730233582Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.730289774Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.730429957Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.730632783Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.730817338Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731048945Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73124454Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73125529Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731317082Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovnkube-node-9fjdd.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731485117Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-node/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731494867Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731499867Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731508127Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731683112Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731750084Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731789465Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731798435Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731803335Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.731833666Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73198924Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732056032Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732076383Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/nbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732096633Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/nbdb/nbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732102363Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/nbdb/nbdb/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732133754Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/nbdb/nbdb/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732269708Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73234014Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73235703Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/northd/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732366641Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/northd/northd/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732373121Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/northd/northd/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732421772Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/northd/northd/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732532685Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732584317Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/northd/northd/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732630788Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-acl-logging/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732641248Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732645288Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73269581Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732795193Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732885095Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732908226Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732919786Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-controller/ovn-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732927846Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.732962417Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.733218794Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.733286696Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.733331157Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovnkube-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.733340708Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.733351988Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.733372418Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735068345Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735152787Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73525979Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/sbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73526975Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/sbdb/sbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735276871Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/sbdb/sbdb/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735285591Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/sbdb/sbdb/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735399474Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735467606Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-9fjdd/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735485596Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735569519Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovnkube-node-dvlk7.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735722943Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-node/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735737283Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735744224Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.735754954Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73597216Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736040332Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736072013Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736078463Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736083273Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736149225Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736281498Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73634869Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736366681Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/nbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736377471Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/nbdb/nbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736389591Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/nbdb/nbdb/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736429702Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/nbdb/nbdb/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736527355Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736592947Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736611278Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/northd/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736626628Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/northd/northd/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736634268Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/northd/northd/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736679749Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/northd/northd/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736790682Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736853104Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/northd/northd/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736918886Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovn-acl-logging/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736931946Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736936296Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.736942156Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.737041179Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.737095871Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovn-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.737106941Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovn-controller/ovn-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.737111721Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.737120201Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.737478061Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.737544693Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.737587144Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovnkube-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.737597374Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.737601325Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.737629595Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.739465216Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.739535918Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.739565979Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/sbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.739576849Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/sbdb/sbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.739582969Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/sbdb/sbdb/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.73961666Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/sbdb/sbdb/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.739716863Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.739779134Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dvlk7/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.739813835Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.739912968Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovnkube-node-rmj6f.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740058452Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-node/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740068282Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740073833Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740104954Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740238217Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740303509Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74034269Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.7403551Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74036013Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740376001Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740518695Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740608257Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740625378Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/nbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740632028Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/nbdb/nbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740638578Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/nbdb/nbdb/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74068761Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/nbdb/nbdb/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740788462Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740863324Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740923626Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/northd/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740936236Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/northd/northd/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740940186Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/northd/northd/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.740965217Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/northd/northd/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74106946Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741142982Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/northd/northd/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741158532Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovn-acl-logging/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741164853Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741168723Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741220494Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741314757Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741357098Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovn-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741366078Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovn-controller/ovn-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741370828Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741395499Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741774929Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741853202Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741893833Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovnkube-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741903193Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741907943Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.741934024Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.743804535Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74398102Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744002861Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/sbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744021291Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/sbdb/sbdb/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744035002Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/sbdb/sbdb/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744071532Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/sbdb/sbdb/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744168155Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744235317Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-rmj6f/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744258478Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744315049Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74434784Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74435634Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744406382Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744502335Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744626458Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744672559Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74471754Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744800063Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744916016Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.744977288Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.745029209Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74506613Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.7450726Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74507767Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.745123761Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.745172973Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.745181363Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.745221074Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.7454162Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.745645866Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74578836Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746012266Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746234762Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746279583Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746346605Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746388426Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746425117Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746469989Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746546371Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746661774Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746753866Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746796407Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746832428Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74690044Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.746950742Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.747034044Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.747077125Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.747111996Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.747226569Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.747282961Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.747291181Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.747325782Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.747385714Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.747422605Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.747531668Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74761662Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.747912618Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74799249Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748122394Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748243097Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748449043Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748508254Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748540186Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748601717Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748642058Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748683029Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74871942Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748802033Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748913166Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748946197Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.748982478Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.7490476Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.74908627Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.749136912Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.749172533Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.749222424Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.749230574Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.749256145Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.749411839Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.749594864Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.749730238Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.749952714Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.75016936Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.75017948Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-57cdc4b768-46bsl/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750211141Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-57cdc4b768-46bsl/service-ca-operator-57cdc4b768-46bsl.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750299574Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-57cdc4b768-46bsl/service-ca-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750312484Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-57cdc4b768-46bsl/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750316704Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-57cdc4b768-46bsl/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750340385Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-57cdc4b768-46bsl/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.75052026Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-57cdc4b768-46bsl/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750684094Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750716185Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750768107Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750802948Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750852889Z namespaces/openshift-service-ca/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750914591Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750959692Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.750995033Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751054704Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751083095Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751176028Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.75127299Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751371603Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751414314Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751458766Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751505117Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751536228Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.75162394Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751685072Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751749023Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751834266Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751938239Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.751946319Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.75199011Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.752047042Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.752088013Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.752195006Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.752270208Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.752386291Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.752469643Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.752581366Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.75270059Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.752896765Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.752963577Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753012118Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753053049Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753100411Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753150962Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753191153Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753268195Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753355258Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753373428Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.75342583Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753477201Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753515892Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753559383Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753572384Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753619905Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.753852041Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.754059087Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.754201371Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.754389576Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.754575191Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.754589272Z namespaces/openshift-service-ca/pods/service-ca-55f6fc7b86-pmkrf/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.754599122Z namespaces/openshift-service-ca/pods/service-ca-55f6fc7b86-pmkrf/service-ca-55f6fc7b86-pmkrf.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.754689924Z namespaces/openshift-service-ca/pods/service-ca-55f6fc7b86-pmkrf/service-ca-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.754700595Z namespaces/openshift-service-ca/pods/service-ca-55f6fc7b86-pmkrf/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.754705385Z namespaces/openshift-service-ca/pods/service-ca-55f6fc7b86-pmkrf/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.754713245Z namespaces/openshift-service-ca/pods/service-ca-55f6fc7b86-pmkrf/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.754975502Z namespaces/openshift-service-ca/pods/service-ca-55f6fc7b86-pmkrf/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755046794Z namespaces/openshift-service-ca/pods/service-ca-55f6fc7b86-pmkrf/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755103506Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755139967Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755208599Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.75525116Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755292481Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755336482Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755403824Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755444935Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755483436Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755515167Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.75560923Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755685682Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755765344Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755803305Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755835086Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755910548Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.755946439Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756045042Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756077533Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756125374Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756218546Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756263758Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756318209Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756412432Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756490424Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756571356Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756649558Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756727531Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.756860074Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757019028Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757056229Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757106921Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757141802Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757192883Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757232544Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757280186Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757396279Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757476321Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757511872Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757560703Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757595854Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757645536Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757680797Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757688747Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757732358Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.757902333Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.758102058Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.758242122Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.758439938Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.758622672Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.758640603Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.758700975Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.758746296Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.758793587Z namespaces/openshift/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.758851619Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.758910011Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.758940891Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759002913Z namespaces/openshift/apps/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759033194Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759122596Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.75927012Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759354603Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759399024Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759461046Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759524657Z namespaces/openshift/batch/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759574949Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759654201Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759700122Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759781044Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.759864817Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.760000341Z namespaces/openshift/core/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.760035131Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.760141314Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.760280398Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.76035658Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.760434862Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.760517475Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.760661959Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.760824123Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.760880045Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.760950987Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.760969637Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.761018989Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.762203521Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.762243742Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.762339875Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.762420417Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.762609782Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.762708055Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.762803027Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.763012063Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.763182388Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.763304101Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.763397354Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.763493726Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.76361185Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.763716063Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.763827596Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.763945419Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.764050942Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.764162965Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.764279658Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.764393871Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.764490374Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.764593987Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.764688799Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.764787262Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.764906315Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.765001238Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.765096841Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.765230844Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.765346348Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.76545312Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.765554343Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.765665686Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.765771269Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.765878842Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.765991105Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.766090828Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.766212601Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.766369925Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.766457598Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.766580461Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.766679344Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.766777337Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.766939671Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.767024894Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.767115336Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.7672495Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.767409744Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.767528537Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.767643891Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.767736583Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.767835266Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.768005821Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.768126964Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.768242607Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.76836047Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.768477374Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.768591497Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.768681409Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.768778652Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.768915655Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769001368Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769045279Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769133472Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769208754Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769335117Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769368838Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.76941958Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769461341Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769527632Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769536323Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769572484Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769747689Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.769957174Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.770111059Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.2.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.770305874Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.3.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.770481139Z namespaces/openshift/policy/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.770586552Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.770650903Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.770682684Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.770746056Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.770755286Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.770805047Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.770953142Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.771072285Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.771178718Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.771293281Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.771463566Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.771569108Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.771680542Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.771820186Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.771963239Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.772077022Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.772184705Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.772319079Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.772455443Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.772560686Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.772663799Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.772775532Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.772913796Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.773027089Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.773131611Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.773234934Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.773329177Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.77343903Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.773546543Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.773652136Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.773751879Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.773860151Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.773978115Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.774119949Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.774229152Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.774328455Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.774448378Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.77453522Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.774657964Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.774783927Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.77490752Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.775013633Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.775128486Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.77525321Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.775358983Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.775464155Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.775587559Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.775718702Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.775854476Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.776040661Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.776170975Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.776325399Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.776462653Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.776580586Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.776699569Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.776778102Z network_logs/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.776822483Z network_logs/cluster_scale [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.776929946Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.777003328Z network_logs/multi-networkpolicy [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.77707923Z network_logs/net-attach-def [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.777168632Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.777315566Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.77743169Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.780485964Z nodes/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.780508154Z nodes/debug [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.780605967Z nodes/ip-10-0-130-79.ec2.internal/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.780645538Z nodes/ip-10-0-130-79.ec2.internal/cpu_affinities.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.780966087Z nodes/ip-10-0-130-79.ec2.internal/dmesg [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.781246434Z nodes/ip-10-0-130-79.ec2.internal/ethtool_channels [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.781359558Z nodes/ip-10-0-130-79.ec2.internal/ethtool_features [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.78144634Z nodes/ip-10-0-130-79.ec2.internal/ip-10-0-130-79.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.781982075Z nodes/ip-10-0-130-79.ec2.internal/irq_affinities.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.78215976Z nodes/ip-10-0-130-79.ec2.internal/lscpu [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.782270253Z nodes/ip-10-0-130-79.ec2.internal/lspci [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.782381436Z nodes/ip-10-0-130-79.ec2.internal/podresources.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.782483828Z nodes/ip-10-0-130-79.ec2.internal/pods_info.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.782614712Z nodes/ip-10-0-130-79.ec2.internal/proc_cmdline [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.782690344Z nodes/ip-10-0-130-79.ec2.internal/sysinfo.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.784473933Z nodes/ip-10-0-130-79.ec2.internal/sysinfo.tgz [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.784642968Z nodes/ip-10-0-141-13.ec2.internal/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.78471781Z nodes/ip-10-0-141-13.ec2.internal/cpu_affinities.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.784993097Z nodes/ip-10-0-141-13.ec2.internal/dmesg [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.785262545Z nodes/ip-10-0-141-13.ec2.internal/ethtool_channels [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.785325637Z nodes/ip-10-0-141-13.ec2.internal/ethtool_features [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.785409589Z nodes/ip-10-0-141-13.ec2.internal/ip-10-0-141-13.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.785910893Z nodes/ip-10-0-141-13.ec2.internal/irq_affinities.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.786066557Z nodes/ip-10-0-141-13.ec2.internal/lscpu [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.786190001Z nodes/ip-10-0-141-13.ec2.internal/lspci [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.786280113Z nodes/ip-10-0-141-13.ec2.internal/podresources.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.786415277Z nodes/ip-10-0-141-13.ec2.internal/pods_info.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.78653083Z nodes/ip-10-0-141-13.ec2.internal/proc_cmdline [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.786611332Z nodes/ip-10-0-141-13.ec2.internal/sysinfo.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.788281958Z nodes/ip-10-0-141-13.ec2.internal/sysinfo.tgz [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.788434922Z nodes/ip-10-0-142-209.ec2.internal/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.788453513Z nodes/ip-10-0-142-209.ec2.internal/cpu_affinities.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.788794472Z nodes/ip-10-0-142-209.ec2.internal/dmesg [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.78908424Z nodes/ip-10-0-142-209.ec2.internal/ethtool_channels [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.789166062Z nodes/ip-10-0-142-209.ec2.internal/ethtool_features [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.789247454Z nodes/ip-10-0-142-209.ec2.internal/ip-10-0-142-209.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.789864431Z nodes/ip-10-0-142-209.ec2.internal/irq_affinities.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.790010086Z nodes/ip-10-0-142-209.ec2.internal/lscpu [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.790141859Z nodes/ip-10-0-142-209.ec2.internal/lspci [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.790229481Z nodes/ip-10-0-142-209.ec2.internal/podresources.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.790340175Z nodes/ip-10-0-142-209.ec2.internal/pods_info.json [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.790466758Z nodes/ip-10-0-142-209.ec2.internal/proc_cmdline [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.79054561Z nodes/ip-10-0-142-209.ec2.internal/sysinfo.log [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.792261267Z nodes/ip-10-0-142-209.ec2.internal/sysinfo.tgz [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.792428182Z pod_network_connectivity_check/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.792472223Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.792576896Z static-pods/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.792587556Z static-pods/kube-apiserver/ [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.799827176Z [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.799857566Z sent 71,980 bytes received 8,882,706 bytes 5,969,790.67 bytes/sec [must-gather] [must-gather-f7kkf] OUT 2026-09-18T23:35:00.799865317Z total size is 128,533,666 speedup is 14.35 [must-gather] [must-gather ] OUT 2026-09-18T23:35:01.104163342Z namespace/openshift-must-gather-s6w2r deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 6efe82c2-d17a-4b5f-be53-03909b4cd2d6 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.34" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-qwmx8 [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-qwmx8-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-qwmx8 [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 17700203 Sep 18 23:37 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-qwmx8/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts d54ff49] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-qwmx8/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 6ef420b..d54ff49 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-09-18T23:37:33.860Z","type":3}]